Apparatuses and methods for registering a real-time image feed from an imaging device to a steerable catheter

ABSTRACT

A method of registering a real-time image feed from an imaging device inserted into a steerable catheter using a navigation system is provided. The method includes inserting the imaging device into a working channel of the steerable catheter and generating a real-time image feed of one or more reference points, wherein the orientation of the reference points is known. The method further includes orienting a handle of the steerable catheter to a neutral position, displaying the real-time image feed on a display of the navigation system, and registering the real-time image feed to the steerable catheter by rotating the displayed image so that the reference points in the real-time image feed are matched to the known orientation of the reference points.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. Ser. No. 15/208,534,filed on Jul. 12, 2016, which in turn is a continuation of U.S. Ser. No.14/259,362, filed on Apr. 23, 2014. Both of these applications arehereby incorporated by reference in their entireties.

BACKGROUND

The invention relates generally to medical devices and particularly toapparatuses and methods associated with a range of image guided medicalprocedures for detecting, sampling, staging and/or treating targettissues in the lungs of a patient.

Image guided surgery (IGS), also known as image guided intervention(IGI), enhances a physician's ability to locate instruments withinanatomy during a medical procedure. IGS can include 2-dimensional (2D),3-dimensional (3D), and 4-dimensional (4D) applications. The fourthdimension of IGS can include multiple parameters either individually ortogether such as time, motion, electrical signals, pressure, airflow,blood flow, respiration, heartbeat, and other patient measuredparameters.

Although significant improvements have been made in these fields, a needremains for improved medical devices and procedures for visualizing,accessing, locating, real-time confirming while sampling andmanipulating a target tissue.

SUMMARY OF THE INVENTION

Among the various aspects of the present invention may be notedapparatuses for use in and methods associated with medical procedures;such apparatuses and methods, for example, may include apparatuses andmethods that enhance a physician's ability to confirm the location of atarget tissue within a patient during a medical procedure, such as imageguided surgery (IGS) or image guided intervention (IGI) and suchapparatuses and methods may further include apparatuses and methods thatfacilitate visualizing, accessing, locating, and manipulating thetargeted tissue.

Briefly, therefore, one aspect of the present invention is a method ofconfirming the location of a target tissue within a patient using anavigation system. The navigation system comprises a localizationdevice, a display, and a pre-acquired image dataset of an airway of thepatient. The navigation system is adapted to display images from theimage dataset and to provide location information of a medical devicewithin the patient in relation to a patient tracking device comprising aplurality of localization elements. The method comprises affixing thepatient tracking device to an external surface of the patient, trackingthe location of the patient tracking device using the navigation system,displaying an image from the image dataset on the display, wherein thedisplayed image is registered to the patient tracking device, anddetermining an initial location of the target tissue in the imagedataset and navigating a steerable catheter through the airway of thepatient to a position proximate the initial location. The steerablecatheter has a proximal end portion and a distal end portion terminatingin a tip, a working channel extending there between, and a localizationelement disposed proximate the distal end portion thereof. The methodfurther comprises tracking the location of the localization element ofthe steerable catheter in the airway using the navigation system,generating information regarding the presence of the target tissue usinga tissue sensing device inserted into the working channel of thesteerable catheter, and determining a confirmed location of the targettissue using the generated information regarding the presence of thetarget tissue and the tracked location of the localization element. Themethod further comprises recording the confirmed location of the targettissue and, displaying the confirmed location of the target tissue onthe display of the navigation system in an image from the image dataset.

Another aspect of the present invention is a method of navigating apercutaneous needle to a target tissue within a patient using anavigation system. The navigation system comprises a localizationdevice, a display, and a pre-acquired image dataset of an airway of thepatient. The navigation system is adapted to display images from theimage dataset and to provide location information of a medical devicewithin the patient in relation to a patient tracking device comprising aplurality of localization elements. The method comprises affixing thepatient tracking device to an external surface of the patient, trackingthe location of the patient tracking device using the navigation systemto monitor the respiratory state of the patient, and displaying an imagefrom the image dataset on the display as a function of the monitoredrespiratory state, wherein the displayed image is registered to thepatient tracking device. The method further comprises determining aninitial location of the target tissue in the image dataset andnavigating a steerable catheter through the airway of the patient to aposition proximate the initial location. The steerable catheter has aproximal end portion and a distal end portion terminating in a tip, aworking channel extending there between, and a localization elementdisposed proximate the distal end portion thereof. The method furthercomprises tracking the location of the localization element of thesteerable catheter in the airway using the navigation system, generatingone or more images of the target tissue using an imaging device insertedinto the working channel of the steerable catheter, and determining aconfirmed location of the target tissue in relation to the patienttracking device using the generated images and the tracked location ofthe localization element. The method further comprises recording theconfirmed location of the target tissue, the recording comprisingfour-dimensional data comprising a three-dimensional location of theconfirmed target tissue in relation to the patient tracking device andthe respiratory state of the patient at the time the location of thetarget tissue was confirmed and applying the confirmed location of thetarget tissue to an image from the image dataset depicting the airway atthe respiratory state of the patient at the time the location of thetarget tissue was confirmed. The method further comprises displaying theconfirmed location of the target tissue on the display of the navigationsystem in an image from the image dataset, the displayed image depictingthe airway at the respiratory state of the patient at the time thelocation of the target tissue was confirmed. Furthermore, the methodcomprises displaying a trajectory of a percutaneous device from an entrypoint on the patient's body to the confirmed location on the display ofthe navigation system, wherein the percutaneous device includes alocalization element, inserting the percutaneous device into the patientand navigating the percutaneous device to the confirmed location, andintercepting the target tissue at the confirmed location.

Another aspect of the present invention is a method of navigating amedical device to the confirmed location of the target tissue usingindicia indicating the confirmed location of the target tissue and/orindicia indicating the location of the medical device. Thus the methodmay include displaying the confirmed location of the target tissue onthe display of the navigation system without requiring that an image ofthe image dataset be displayed. This method of navigating a medicaldevice to the confirmed location of the target tissue does not requirere-registering one or more image datasets to the patient so long as thepatient tracking device affixed to the patient does not move or thepatient does not move relative to an electromagnetic field generator ofthe navigation system. Therefore, this method does not requiredisplaying a hybrid “Inspiration-Expiration” 3D airway model, one ormore images from one or more image datasets, a navigation pathway,and/or a real-time image feed from a bronchoscopic video camera, inorder to permit a physician or other healthcare professional innavigating a medical device to the confirmed location of the targettissue.

Another aspect of the present invention is directed to a method ofregistering a real-time image feed from an imaging device inserted intoa steerable catheter using a navigation system comprising a display. Thesteerable catheter comprises an elongate flexible shaft having aproximal end portion, a distal end portion terminating in a tip, aworking channel extending therebetween, and handle attached to theproximal end portion of the flexible shaft. The method comprisesinserting the imaging device into the working channel of the steerablecatheter, generating a real-time image feed of one or more referencepoints, wherein the orientation of the reference points is known,orienting the handle of the steerable catheter to a neutral position,displaying the real-time image feed on the display, and registering thereal-time image feed to the steerable catheter by rotating the displayedimage so that the reference points in the real-time image feed arematched to the known orientation of the reference points.

Another aspect of the present invention is directed to a method ofenhancing registration of the real-time image feed of a bronchoscopicvideo camera by correcting image distortion in the real time image feed.For example, bronchoscopic video cameras typically include fish-eyelenses which increase the field of view of the bronchoscopic videocamera thus providing the physician or other healthcare professionalwith a larger view of the airway of the patient. However, the fish-eyelenses introduce barrel distortion into the real-time image feed. Due tothis barrel distortion, the interpretation of the real-time image feedmay be compromised. Correcting for this image distortion in thereal-time image feed provides a more accurate depiction of the airway ofthe patient, thus permitting an enhanced registration of the real-timeimage feed.

Yet another aspect of the present invention is directed to theconstruction and use of a hybrid “Inspiration-Expiration” 3D airwaymodel. The hybrid “Inspiration-Expiration” 3D airway model may be usedto reduce or eliminate errors in registration. Constructing the hybrid“Inspiration-Expiration” 3D airway model comprises calculating apopulation of deformation vector fields, wherein the deformation vectorfield(s) comprise vectors from one or more voxels in inspiration imagesor in an inspiration 3D airway model to one or more corresponding voxelsin expiration images or in an expiration 3D airway model. After thedeformation vector field is calculated, the inspiration images and/orthe inspiration 3D airway model may be deformed to the expiration stateof the patient using the deformation vector field. Accordingly, thevoxels in the inspiration images and/or inspiration 3D airway model aredeformed to match the location, shape, and orientation of the airways ofthe patient at expiration. This results in the hybrid“Inspiration-Expiration” 3D airway model, wherein the hybrid“Inspiration-Expiration” 3D airway model contains the structuralinformation of the airways of patient depicted in the inspiration imagesand/or inspiration 3D airway model. However, this structural informationis now more closely matched to the location, shape, and orientation ofthe airways of the patient depicted in the expiration images and/orexpiration 3D airway model. Accordingly, the deformation vectorsrepresent a change in location of the structure of the airway and achange in shape of the structure of the airway from inspiration toexpiration.

Yet another aspect of the present invention is directed to a method ofinjecting dye into a target tissue using a needle inserted into theworking channel of a steerable catheter or using a needle inserted intothe working channel of a percutaneous needle. Thus, when sampling thetarget tissue using a medical device inserted into the steerablecatheter or percutaneous needle, the presence of dye in the sampleprovides an indication that the correct target tissue was sampled. Thismay be helpful, for example, in lung resections where there issignificant movement of the lungs of the patient. For example, duringlung resections there may be a gap between the chest wall and the lungand the physician or other healthcare profession may use a rigid scopeto enter into the patient. Because the target tissue was previously dyedusing a needle inserted into the working channel of steerable catheteror using a needle inserted into the working channel of the percutaneousneedle, the physician or other healthcare professional may be able tovisually see the dye. This may assist the physician or healthcareprofessional in sampling and/or treating the correct target tissue.

Yet another aspect of the present invention is directed to a method ofsimulating and/or displaying a variety of image views using a navigationsystem based on the position and orientation (POSE) of a localizationelement in a steerable catheter, percutaneous device, and/or some othermedical device. For example, the navigation system may be able tosimulate and/or display axial images, coronal images, oblique images,orthogonal image slices, oblique or off-axis image slices, volumerendered images, segmented images, fused modality images, maximumintensity projection (MIPS) images, video, and video enhanced images. Tosimulate these views, the navigation system may modify one or moreimages from an image dataset using known image manipulation techniques.The images in the image dataset may be fluoroscopic images, ultrasoundimages, to computed tomography (CT) images, fused computedtomography—positron emission tomography (CT/PET) images, magneticresonance imaging (MRI) images, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of the present invention, both as to its construction andoperation can best be understood with reference to the accompanyingdrawings, in which like numerals refer to like parts, and in which:

FIG. 1 left perspective view of a patient tracking device on a patientaccording to an embodiment of the invention;

FIG. 2 is a schematic illustration of an image analysis system accordingto an embodiment of the invention;

FIG. 3 is a schematic illustration of a navigation system according toan embodiment of the invention;

FIG. 4 is a graphical representation illustrating the function of thepatient tracking device according to an embodiment of the invention;

FIG. 5A is an illustration of a patient being imaged using an imagingdevice according to an embodiment of the invention;

FIG. 5B is an illustration of a patient being imaged using an imagingdevice according to an embodiment of the invention;

FIG. 5C is a schematic illustration of an image dataset according to anembodiment of the invention;

FIG. 6A is a schematic illustration of an inspiration 3D airway modelaccording to an embodiment of the invention;

FIG. 6B is a schematic illustration of an expiration 3D airway modelaccording to an embodiment of the invention;

FIG. 6C is a schematic illustration of a hybrid “Inspiration-Expiration”3D airway model according to an embodiment of the invention;

FIG. 7 is a front perspective view of a hybrid “Inspiration-Expiration”3D airway model according to an embodiment of the invention;

FIG. 8 is a schematic illustrating vector distances of the patienttracking device according to an embodiment of the invention;

FIG. 9A is a schematic illustrating vector distances from a localizationelement on the patient tracking device according to an embodiment of theinvention;

FIG. 9B is a schematic illustrating vector distances from an imagedataset according to an embodiment of the invention;

FIG. 10 is a flowchart illustrating a method according to an embodimentof the invention;

FIG. 11 is a left side view of a steerable catheter according to anembodiment of the invention;

FIG. 11A is a left partial section view of a steerable catheteraccording to an embodiment of the invention;

FIG. 12A is a left partial cut away view of a steerable catheteraccording to an embodiment of the invention;

FIG. 12B is a left partial cut away view of a steerable catheteraccording to an embodiment of the invention;

FIG. 13 is a left side view of a percutaneous needle according to anembodiment of the invention;

FIG. 13A is a left partial cut away view of a percutaneous needleaccording to an embodiment of the invention;

FIG. 14 illustrates a population of images which may be displayed on adisplay of a navigation system according to an embodiment of theinvention;

FIG. 15 is a flowchart illustrating a method of registering thereal-time image feed from a bronchoscopic video camera to a steerablecatheter according to an embodiment of the invention;

FIG. 16 is a left perspective view of a steerable catheter in a jig forregistering the real-time image feed from a bronchoscopic video camerato a steerable catheter according to an embodiment of the presentinvention;

FIG. 16A is a front view of a steerable catheter in a jig forregistering the real-time image feed from a bronchoscopic video camerato a steerable catheter according to an embodiment of the presentinvention;

FIG. 16B is an image from a real-time image feed from a non-registeredbronchoscopic video camera in a steerable catheter according to anembodiment of the present invention;

FIG. 16C is an image from a real-time image feed from a registeredbronchoscopic video camera in a steerable catheter according to anembodiment of the present invention;

FIG. 17 is a flowchart illustrating a method of registering thereal-time image feed from a bronchoscopic video camera to a steerablecatheter according to an embodiment of the invention;

FIG. 17A is a flowchart illustrating additional steps of a method ofregistering the real-time image feed from a bronchoscopic video camerato a steerable catheter according to an embodiment of the invention;

FIG. 18A is an image of an expected orientation of anatomical featuresin the airway of the patient according to an embodiment of theinvention;

FIG. 18B is an image from a real-time image feed from a non-registeredbronchoscopic video camera in a steerable catheter according to anembodiment of the present invention;

FIG. 18C is an image from a real-time image feed from a registeredbronchoscopic video camera in a steerable catheter according to anembodiment of the present invention;

FIG. 19 is a section view of steerable catheter illustrating registeringthe real-time image feed from a bronchoscopic video camera to asteerable catheter and to a localization element of the steerablecatheter according to an embodiment of the invention;

FIG. 20A is a flowchart illustrating a portion of a method of confirmingthe location of a target tissue according to an embodiment of theinvention;

FIG. 20B is a flowchart illustrating a portion of a method of confirmingthe location of a target tissue according to an embodiment of theinvention;

FIG. 20C is a flowchart illustrating a portion of a method of confirmingthe location of a target tissue according to an embodiment of theinvention;

FIG. 20D is a flowchart illustrating a portion of a method of confirmingthe location of a target tissue according to an embodiment of theinvention;

FIG. 20E is a flowchart illustrating a portion of a method of confirmingthe location of a target tissue according to an embodiment of theinvention;

FIG. 21 is an image from an endobronchial ultrasound device according toan embodiment of the invention;

FIG. 22 illustrates a population of images which may be displayed on adisplay of a navigation system according to an embodiment of theinvention;

FIG. 22A illustrates an enlarged view of an image which may be displayedon a display of a navigation system according to an embodiment of theinvention; and

FIG. 23 illustrates a population of images which may be displayed on adisplay of a navigation system according to an embodiment of theinvention.

DETAILED DESCRIPTION

The accompanying Figures and this description depict and describeembodiments of a navigation system (and related methods and devices) inaccordance with the present invention, and features and componentsthereof. It should also be noted that any references herein to front andback, right and left, top and bottom and upper and lower are intendedfor convenience of description, not to limit the present invention orits components to any one positional or spatial orientation.

Those of skill in the art will appreciate that in the detaileddescription below, certain well known components and assembly techniqueshave been omitted so that the present methods, apparatuses, and systemsare not obscured in unnecessary detail.

With larger volumes of patients expected to obtain lung cancerscreening, obtaining definitive diagnoses may avoid numerous unneededlung resections as about only 4% of patients from lung cancer screeningare typically found to have a malignancy. However, peripheral targettissues (e.g., nodule, lesion, lymph node, tumor, etc.) that are smallerthan 2 cm in size still present a difficult problem to solve. Typicalbronchoscopes that are designed mainly for central airway inspectionwill be limited to the extent they can travel due to their largediameters before becoming wedged in the airway of the patient. Thus, toaffect the 5 and 10 year survival rate of patient's that have targettissues which may be less than 2 cm in size, the apparatuses and methodsas described herein allow for enhanced target tissue analysis forstaging, intercepting target tissues in the periphery of the lungs thatmay not be accessible via airways, obtaining larger and higher qualitytissue samples for testing, and provide a streamlined patient flow.Accordingly, the apparatuses and methods described herein enable aphysician or other healthcare professional to initially determine thelocation of a target tissue and to confirm the location of the targettissue. In one embodiment, a hybrid “Inspiration-Expiration” 3D modelmay be used to provide patient specific 4D respiratory models whichaddress peripheral respiratory motion. In certain patients, portions ofthe lungs including the upper lobes may move, on average, 15 mm betweeninspiration and expiration. Using a steerable catheter with an imagingdevice, such as a radial endobronchial ultrasound (EBUS) device insertedtherein, a physician or other healthcare professional can determine aconfirmed location of the target tissue. Additionally, apparatuses andmethods described herein enable a physician or other healthcareprofessional to transition to a percutaneous approach to the targettissue, if needed. If the physician or other healthcare professional isunable to reach the target tissue for any reason, including but notlimited to, the target tissue being below the surface of the airway(i.e., sub-surface target tissue), no airway proximate the targettissue, the pathway to the target tissue is very tortuous, or larger oradditional tissue sample from a core biopsy is desired, the physician orother healthcare professional may insert navigated percutaneous needlesto the confirmed location of the target tissue. Thus it will beunderstood that the apparatuses and methods described herein may be usedto intercept target tissue(s) in the airway, on the wall of the airway,in the wall of the airway, and/or beyond the wall of the airway. Thatis, the apparatuses and methods described herein may be used tointercept target tissue(s) not only inside the airway, but may intercepttarget tissue(s) and other anatomical structures inside and/or beyondthe wall of the airway. Thus in certain embodiments, sub-surface targettissue(s) may be intercepted.

Additionally, the apparatuses and methods described herein provide easyto understand localization information to the physician or otherhealthcare professional, as well as display the preferred entry site andtrajectory views of the percutaneous needle that are aligned to thetarget tissue. Once aligned, the physician or other healthcareprofessional may direct the percutaneous needle along the trajectory tothe target tissue while viewing a display of the location of the tip ofpercutaneous needle on a navigation system as described herein. Thephysician or other healthcare professional may then intercept the targettissue in a variety of ways, including, but not limited to, performing astandard core biopsy, an aspiration, and/or delivering therapy using avariety of medical devices inserted through the percutaneous needle.

As shown in FIG. 1 , an apparatus according to an embodiment of theinvention includes patient tracking device (PTD) 20 comprising two ormore markers 22 and two or more localization elements 24 proximatemarkers 22. Markers 22 are visible in images captured by an imagingdevice and the position and orientation (POSE) of localization elements24 may be tracked by a localization device in an image analysis systemand/or a navigation system. PTD 20 comprises a population of separatepads 26, 28, 30, each of which may include one or more markers 22 andlocalization elements 24 proximate markers 22. First and second pads 26,28 may each include one marker 22 and one localization element 24. Thirdpad 30 may include four markers 22 and four localization elements 24located proximate the periphery of third pad 30. Additionally, wires 32,34, 36 are used to connect localization elements 24 in each of first,second, and third pads 26, 28, 30 to image analysis system 50 (see FIG.2 ) and/or navigation system 70 (see FIG. 3 ). In alternativeembodiments, localization elements 24 may be wirelessly connected tonavigation system 70. FIG. 1 illustrates PTD 20 having six markers 22and six localization elements 24, but any number of two or more markers22 and localization elements 24 can be used. Patient tracking device(PTD) 20 can be coupled to a dynamic body such as, for example, aselected dynamic portion of the anatomy of a patient 10.

Markers 22 are constructed of a material that can be viewed on an image,such as, for example, X-ray images or CT images. In certain embodiments,markers 22 can be, for example, radiopaque such that they are visiblevia fluoroscopic imaging. In other embodiments, for example, markers 22may be echogenic such that they are visible via ultrasonic imaging. Inyet other embodiments, markers 22 may be both radiopaque and echogenic.In certain embodiments, for example, localization elements 24 comprisesix (6) degree of freedom (6DOF) electromagnetic coil sensors. In otherembodiments, localization elements 24 comprise five (5) degree offreedom (5DOF) electromagnetic coil sensors. In other embodiments,localization elements 24 comprise other localization devices such asradiopaque markers that are visible via fluoroscopic imaging andechogenic patterns that are visible via ultrasonic imaging. In yet otherembodiments, localization elements 24 can be, for example, infraredlight emitting diodes, and/or optical passive reflective markers.Localization elements 24 can also be, or be integrated with, one or morefiber optic localization (FDL) devices.

While PTD 20 is shown comprising a population of separate padscontaining markers 22 and localization elements 24, in certainembodiments, PTD 20 may comprise one pad containing markers 22 andlocalization elements 24. In another embodiment, for example, PTD 20 mayinclude markers 22 but not localization elements 24. In anotherembodiment, for example, PTD 20 may include localization elements 24 butnot markers 22. In various embodiments, markers 22 and localizationelements 24 can be the same device. In certain embodiments, for example,localization elements 24 may function or serve as markers 22. PTD 20 canbe a variety of different shapes and sizes. For example, in oneembodiment PTD 20 is substantially planar, such as in the form of a padthat can be disposed at a variety of locations on a patient's 10 body.PTD 20 can be coupled to patient 10 with adhesive, straps, hook andpile, snaps, or any other suitable coupling method. In anotherembodiment the PTD can be a catheter type device with a pigtail oranchoring mechanism that allows it to be attached to an internal organor along a vessel.

As described more fully elsewhere herein, an image analysis system isconfigured to receive image data associated with the dynamic bodygenerated during a pre-surgical or pre-procedural first time interval.The image data can include an indication of a position of each ofmarkers 22 for multiple instants in time during the first time interval.Then a navigation system can also receive position data associated withlocalization elements 24 during a second time interval in which asurgical procedure or other medical procedure is being performed. Thenavigation system can use the position data received from localizationelements 24 to determine a distance between the localization elements 24for a given instant in time during the second time interval. Thenavigation system can also use the image data to determine the distancebetween markers 22 for a given instant in time during the first timeinterval. The navigation system can then find a match between an imagewhere the distance between markers 22 at a given instant in time duringthe first time interval is the same or substantially the same as thedistance between localization elements 24 associated with those markers22 at a given instant in time during the medical procedure, or secondtime interval. Additionally, the navigation system can determine asequence of motion of the markers and match this sequence of motion tothe recorded motion of the markers over the complete procedure orsignificant period of time. Distance alone between the markers may notbe sufficient to match the patient space to image space in manyinstances, the system may also determine the direction the markers aremoving and the range and speed of this motion to find the appropriatesequence of motion for a complex signal or sequence of motion by thepatient.

A physician or other healthcare professional can use the images selectedby the navigation system during a medical procedure performed during thesecond time interval. For example, when a medical procedure is performedon a targeted anatomy of a patient, such as a heart or lung, thephysician may not be able to utilize an imaging device during themedical procedure to guide him to the targeted area within the patient.Accordingly, PTD 20 can be positioned or coupled to the patientproximate the targeted anatomy prior to the medical procedure, andpre-procedural images can be taken of the targeted area during a firsttime interval. Markers 22 of PTD 20 can be viewed with the image data,which can include an indication of the position of markers 22 during agiven path of motion of the targeted anatomy (e.g., the heart) duringthe first time interval. Such motion can be due, for example, toinspiration (i.e., inhaling) and expiration (i.e., exhaling) of thepatient, or due to the heart beating. During a medical procedure,performed during a second time interval, such as a procedure on a heartor lung, the navigation system receives data from localization elements24 associated with a position of localization elements 24 at a giveninstant in time during the medical procedure (or second time interval).The distance between selected pairs of markers 22 can be determined fromthe image data and the distance, range, acceleration, and speed betweencorresponding selected pairs of localization elements 24 can bedetermined based on the position and orientation (POSE) data for giveninstants in time. Accordingly, the range of motion and speed of markers22 can be calculated.

Because localization elements 24 are proximate the location of markers22, the distance between a selected pair of localization elements 24 canbe used to determine an intra-procedural distance between the pair ofcorresponding markers 22. An image from the pre-procedural image datataken during the first time interval can then be selected where thedistance between the pair of selected markers 22 in that imagecorresponds with or closely approximates the same distance determinedusing localization elements 24 at a given instant in time during thesecond time interval. This process can be done continuously during themedical procedure, producing simulated real-time, intra-proceduralimages illustrating the orientation and shape of the targeted anatomy asa catheter, sheath, needle, forceps, guidewire, fiducial deliverydevices, therapy device, or similar medical device(s) is/are navigatedto the targeted anatomy. Thus, during the medical procedure, thephysician can view selected image(s) of the targeted anatomy thatcorrespond to and simulate real-time movement of the anatomy. Inaddition, during a medical procedure being performed during the secondtime interval, such as navigating a catheter or other medical device orcomponent thereof to a targeted anatomy, the location(s) of alocalization element (e.g., an electromagnetic coil sensor) coupled tothe catheter during the second time interval can be superimposed on animage of a catheter. The superimposed image(s) of the catheter can thenbe superimposed on the selected image(s) from the first time interval,providing simulated real-time images of the catheter location relativeto the targeted anatomy. This process and other related methods aredescribed in U.S. Pat. No. 7,398,116, entitled Methods, Apparatuses, andSystems Useful in Conducting Image Guided Interventions, filed Aug. 26,2003, which is hereby incorporated by reference.

Referring now to FIGS. 2 and 3 , two systems which may be used duringimage guided surgery are described in detail. The first systemillustrated in FIG. 2 , is image analysis system 50. Image analysissystem 50 is used during generation of a population of images of patient10 during a first time interval, prior to a medical procedure beingperformed on patient 10. The second system, illustrated in FIG. 3 , isnavigation system 70. Navigation system 70 is used during a medicalprocedure performed on patient 10 during a second time interval. As willbe described, imaging system 50 and navigation system 70 may include, invarious embodiments, substantially similar or identical components.Accordingly, image analysis system 50 and navigation system 70 may beable to carry out substantially similar or identical functions. Incertain embodiments, image analysis system 50 and navigation system 70and may comprise a single system. In certain embodiments, for example,image analysis system 50 may also function or serve as a navigationsystem. In certain embodiments, for example, navigation system 70 mayalso function or serve as an image analysis system.

As shown in FIG. 2 , image analysis system 50 comprises a processor 52having memory component 54, input/output (I/O) component 58, andoptional localization device 56. Image analysis system 50 may alsooptionally include display 60, electromagnetic field generator 62,and/or user interface device(s) 64 (e.g., keyboard, mouse).

Image analysis system 50 further includes and/or is in datacommunication with imaging device 40. Imaging device 40 can be, forexample, a computed tomography (CT) device (e.g., respiratory-gated CTdevice, ECG-gated CT device), a magnetic resonance imaging (MRI) device(e.g., respiratory-gated MRI device, ECG-gated MRI device), an X-raydevice, a 2D or 3D fluoroscopic imaging device, and 2D, 3D or 4Dultrasound imaging devices, or any other suitable medical imagingdevice. In one embodiment, for example, imaging device 40 is a computedtomography—positron emission tomography (CT/PET) device that produces afused computed tomography—positron emission tomography (CT/PET) imagedataset. In the case of a two-dimensional imaging device, a populationof two-dimensional images may be acquired and then assembled intovolumetric data (e.g., three-dimensional (3D) image dataset) as is wellknown in the art using a two-dimensional to three-dimensionalconversion. Pre-procedurally during a first time interval, imagingdevice 40 can be used to generate a population of images of patient 10while PTD 20 is coupled to patient 10; wherein the population of imagesdepict the anatomy of patient 10. The anatomy, may include, but is notlimited to, the lungs, heart, liver, kidneys, and/or other organs ofpatient 10. The population of images can be compiled into an imagedataset. As stated above, some or all markers 22 of PTD 20 are visibleon the population of images and provide an indication of a position ofsome or all of markers 22 during the first time interval. The positionof markers 22 at given instants in time through a path of motion ofpatient 10 can be illustrated with the images.

Processor 52 of image analysis system 50 includes a processor-readablemedium storing code representing instructions to cause the processor 52to perform a process. Processor 52 can be, for example, a commerciallyavailable personal computer, or a less complex computing or processingdevice that is dedicated to performing one or more specific tasks. Forexample, processor 52 can be a terminal dedicated to providing aninteractive graphical user interface (GUI) on optional display 60.Processor 52, according to one or more embodiments of the invention, canbe a commercially available microprocessor. Alternatively, processor 52can be an application-specific integrated circuit (ASIC) or acombination of ASICs, which are designed to achieve one or more specificfunctions, or enable one or more specific devices or applications. Inyet another embodiment, processor 52 can be an analog or digitalcircuit, or a combination of multiple circuits.

Additionally, processor 52 can include memory component 54. Memorycomponent 54 can include one or more types of memory. For example,memory component 54 can include a read only memory (ROM) component and arandom access memory (RAM) component. Memory component 54 can alsoinclude other types of memory that are suitable for storing data in aform retrievable by processor 52. For example, electronicallyprogrammable read only memory (EPROM), erasable electronicallyprogrammable read only memory (EEPROM), flash memory, as well as othersuitable forms of memory can be included within the memory component.Processor 52 can also include a variety of other components, such as forexample, coprocessors, graphic processors, etc., depending upon thedesired functionality of the code.

Processor 52 can store data in memory component 54 or retrieve datapreviously stored in memory component 54. The components of processor 52can communicate with devices external to processor 52 by way ofinput/output (I/O) component 58. According to one or more embodiments ofthe invention, I/O component 58 includes a variety of suitablecommunication interfaces. For example, I/O component 58 can include, forexample, wired connections, such as standard serial ports, parallelports, universal serial bus (USB) ports, S-video ports, local areanetwork (LAN) ports, small computer system interface (SCSI) ports, andso forth. Additionally, I/O component 58 can include, for example,wireless connections, such as infrared ports, optical ports, Bluetooth®wireless ports, wireless LAN ports, or the like. Embodiments of imageanalysis system 50 which include display 60, electromagnetic fieldgenerator 62, and/or user interface device(s) 64, such componentscommunicate with processor 52 via I/O component 58.

Processor 52 can be connected to a network, which may be any form ofinterconnecting network including an intranet, such as a local or widearea network, or an extranet, such as the World Wide Web or theInternet. The network can be physically implemented on a wireless orwired network, on leased or dedicated lines, including a virtual privatenetwork (VPN).

As stated above, processor 52 receives the population of images fromimaging device 40. Processor 52 identifies the position of selectedmarkers 22 within the image data or voxel space using varioussegmentation techniques, such as Hounsfield unit thresholding,convolution, connected component, or other combinatory image processingand segmentation techniques. Processor 52 determines a distance anddirection between the position of any two markers 22 during multipleinstants in time during the first time interval, and stores the imagedata, as well as the position and distance data, within memory component54. Multiple images can be produced providing a visual image at multipleinstants in time through the path of motion of the dynamic body.

As stated above, processor 52 can optionally include a receiving deviceor localization device 56 for tracking the location of localizationelements 24 of PTD 20, as described more fully elsewhere herein. Bytracking localization elements 24 associated with PTD 20 when thepopulation of images are generated by imaging device 40, the populationof images may be gated. That is, image analysis system 50 determines therespiratory phase at which the population of images were generated andthis information may be stored in an image dataset and/or in anotherdata store in memory component 54.

In general, image analysis system 50 may comprise any tracking systemtypically employed in image guided surgery, including but not limitedto, an electromagnetic tracking system. An example of a suitableelectromagnetic tracking subsystem is the AURORA electromagnetictracking system, commercially available from Northern Digital Inc.(Waterloo, Ontario Canada). In one embodiment, image analysis system 50may include an electromagnetic tracking system, typically comprising anelectromagnetic (EM) field generator 62 that emits a series ofelectromagnetic fields designed to engulf patient 10, and localizationelements 24 coupled to PTD 20. In certain embodiments, for example,localization elements 24 are electromagnetic coils that receive aninduced voltage from electromagnetic (EM) field generator 62, whereinthe induced voltage is monitored and translated by localization device56 into a coordinate position of localization elements 24. In certainembodiments, localization elements 24 are electrically coupled totwisted pair conductors to provide electromagnetic shielding of theconductors. This shielding prevents voltage induction along theconductors when exposed to the magnetic flux produced by theelectromagnetic field generator.

Accordingly, localization device 56 can be, for example, an analog todigital converter that measures voltages induced onto localizationelements 24 in the field generated by EM field generator 62; creates adigital voltage reading; and maps that voltage reading to a metricpositional measurement based on a characterized volume of voltages tomillimeters from electromagnetic field generator 62. Position dataassociated with localization elements 24 can be transmitted or sent tolocalization device 56 continuously during imaging of patient 10 duringthe first time interval. Thus, the position of localization elements 24can be captured at given instants in time during the first timeinterval. Because localization elements 24 are proximate markers 22,localization device 56 uses the position data of localization elements24 to deduce coordinates or positions associated with markers 22 duringthe first time interval. The distance, range, acceleration, and speedbetween one or more selected pairs of localization elements 24 (andcorresponding markers 22) is then determined and various algorithms areused to analyze and compare the distance between selected elements 24 atgiven instants in time, to the distances between and orientation amongcorresponding markers 22 observed in a population of pre-proceduralimages.

As shown in FIG. 3 , navigation system 70 comprises a processor 72having memory component 74, input/output (I/O) component 78, andlocalization device 76. Navigation system 70 also includes display 80,electromagnetic field generator 82, and/or user interface device(s) 84(e.g., keyboard, mouse). In certain embodiments, navigation system 50further includes and/or is in data communication with imaging device 40(see FIG. 2 ).

Processor 72 of navigation system 70 includes a processor-readablemedium storing code representing instructions to cause the processor 72to perform a process. Processor 72 can be, for example, a commerciallyavailable personal computer, or a less complex computing or processingdevice that is dedicated to performing one or more specific tasks. Forexample, processor 72 can be a terminal dedicated to providing aninteractive graphical user interface (GUI) on optional display 80.Processor 72, according to one or more embodiments of the invention, canbe a commercially available microprocessor. Alternatively, processor 72can be an application-specific integrated circuit (ASIC) or acombination of ASICs, which are designed to achieve one or more specificfunctions, or enable one or more specific devices or applications. Inyet another embodiment, processor 72 can be an analog or digitalcircuit, or a combination of multiple circuits.

Additionally, processor 72 can include memory component 74. Memorycomponent 74 can include one or more types of memory. For example,memory component 74 can include a read only memory (ROM) component and arandom access memory (RAM) component. Memory component 74 can alsoinclude other types of memory that are suitable for storing data in aform retrievable by processor 72. For example, electronicallyprogrammable read only memory (EPROM), erasable electronicallyprogrammable read only memory (EEPROM), flash memory, as well as othersuitable forms of memory can be included within the memory component.Processor 72 can also include a variety of other components, such as forexample, coprocessors, graphic processors, etc., depending upon thedesired functionality of the code.

Processor 72 can store data in memory component 74 or retrieve datapreviously stored in memory component 74. The components of processor 72can communicate with devices external to processor 72 by way ofinput/output (I/O) component 78. According to one or more embodiments ofthe invention, I/O component 78 includes a variety of suitablecommunication interfaces. For example, I/O component 78 can include, forexample, wired connections, such as standard serial ports, parallelports, universal serial bus (USB) ports, S-video ports, local areanetwork (LAN) ports, small computer system interface (SCSI) ports, andso forth. Additionally, I/O component 78 can include, for example,wireless connections, such as infrared ports, optical ports, Bluetooth®wireless ports, wireless LAN ports, or the like. Additionally, display80, electromagnetic field generator 82, and/or user interface device(s)84, communicate with processor 72 via I/O component 78.

Processor 72 can be connected to a network, which may be any form ofinterconnecting network including an intranet, such as a local or widearea network, or an extranet, such as the World Wide Web or theInternet. The network can be physically implemented on a wireless orwired network, on leased or dedicated lines, including a virtual privatenetwork (VPN).

In general, navigation system 70 may comprise any tracking systemtypically employed in image guided surgery, including but not limitedto, an electromagnetic tracking system. An example of a suitableelectromagnetic tracking subsystem is the AURORA electromagnetictracking system, commercially available from Northern Digital Inc.(Waterloo, Ontario Canada). In one embodiment, navigation system 70 mayinclude an electromagnetic tracking system, typically comprising anelectromagnetic (EM) field generator 82 that emits a series ofelectromagnetic fields designed to engulf patient 10, and localizationelements 24 coupled to PTD 20. In certain embodiments, for example,localization elements 24 are electromagnetic coils that receive aninduced voltage from electromagnetic (EM) field generator 82, whereinthe induced voltage is monitored and translated by localization device76 into a coordinate position of localization elements 24. In certainembodiments, localization elements 24 are electrically coupled totwisted pair conductors to provide electromagnetic shielding of theconductors. This shielding prevents voltage induction along theconductors when exposed to the magnetic flux produced by theelectromagnetic field generator.

Accordingly, localization device 76 may be, for example, an analog todigital converter that measures voltages induced onto localizationelements 24 in the field generated by EM field generator 82; creates adigital voltage reading; and maps that voltage reading to a metricpositional measurement based on a characterized volume of voltages tomillimeters from electromagnetic field generator 82. Position dataassociated with localization elements 24 may be transmitted or sent tolocalization device 76 continuously during the medical procedureperformed during the second time interval. Thus, the position oflocalization elements 24 may be captured at given instants in timeduring the second time interval. Because localization elements 24 areproximate markers 22, localization device 76 uses the position data oflocalization elements 24 to deduce coordinates or positions associatedwith markers 22 during the second time interval. The distance, range,acceleration, and speed between one or more selected pairs oflocalization elements 24 (and corresponding markers 22) is thendetermined and various algorithms are used to analyze and compare thedistance between selected elements 24 at given instants in time, to thedistances between and orientation among corresponding markers 22observed in a population of pre-procedural images.

Because localization elements 24 of PTD 20 may be tracked continuouslyduring the first and/or second time intervals, a sequence of motion ofPTD 20 that represents the motion of an organ of patient 10 or thepatient's 10 respiratory cycle may be collected. As patient 10 inhalesand exhales, the individual localization elements 24 of PTD 20 will moverelative to one another. That is, as patient 10 inhales, the distancebetween some or all of localization elements 24 of PTD 20 may increase.Conversely, as patient 10 exhales, the distance between some or all oflocalization elements 24 of PTD 20 may decrease. The sequence of motionof localization elements 24 is tracked by image analysis system 50and/or navigation system 70 and image analysis system 50 and/ornavigation system 70 derives a respiratory signal based on the positionsof localization elements 24 during the respiratory cycle of patient 10.The sequence of motion may then be analyzed to find unique similarpoints within the image dataset and images within the image dataset maybe grouped.

According to one particular embodiment, the respiratory signal derivedfrom PTD 20 is used to gate the localization information of a medicaldevice in the airway of patient 10. In other embodiments, therespiratory signal derived from PTD 20 is used during the first timeinterval to gate the population of images generated by imaging device40. Using PTD 20 to derive a respiratory signal may assist indetermining multiple airway models, for example, by performing a bestfit of the real-time patient airway model to the image dataset to derivethe optimal registration and gated period in the patient's respiratorycycle. Additionally or alternatively, the respiratory signal may bederived from devices other than PTD 20 that are known in the art formeasuring the respiratory cycle of a patient. In certain embodiments,for example, a device that records the resistance between two locationson the patient may be used to measure the respiratory cycle. Forexample, such device a is similar to a variable potentiometer in thatthe resistance of the patient changes between two fixed points as thepatient inhales and exhales. Thus, the resistance may be measured tocreate a respiratory signal. In other embodiments, a spirometer may beused to measure the respiratory cycle. In yet other embodiments, acardiac signal may be used to gate the localization information of amedical device in the airway of patient 10. It will be understood thatany type of device for generating a cardiac signal may be used,including, but not limited to an ECG device, PTD 20, etc.

FIG. 4 is a schematic illustration indicating how markers 22 of PTD 20can move and change orientation and shape during movement of patient 10.The graph is one example of how the lung volume can change duringinhalation (inspiration) and exhalation (expiration) of patient 10. Thecorresponding changes in shape and orientation of PTD 20 duringinhalation and exhalation are also illustrated. The six markers 22 shownin FIG. 1 are schematically represented and labeled a, b, c, d, e, andf. As described above, a population of images of PTD 20 may be takenduring a first time interval. The population of images include anindication of relative position of one or more markers 22; that is, oneor more markers 22 are visible in the images, and the position of eachmarker 22 is then observed over a period of time. A distance between anytwo markers 22 may then be determined for any given instant of timeduring the first time interval. For example, a distance X betweenmarkers a and b is illustrated, and a distance Y between markers b and fis illustrated. These distances may be determined for any given instantin time during the first time interval from an associated image thatillustrates the position and orientation of markers 22. As illustrated,during expiration of patient 10 at times indicated as A and C, thedistance X is smaller than during inspiration of patient 10, at the timeindicated as B. Likewise, the distance Y is greater during inspirationthan during expiration. The distance between any pair of markers 22 maybe determined and used in the processes described herein. Thus, theabove embodiments are merely examples of possible pair selections. Forexample, a distance between a position of marker e and a position ofmarker b may be determined. In addition, multiple pairs or only one pairmay be selected for a given procedure.

FIGS. 5A and 5B illustrate the generation of a population of imagesduring a first time interval using imaging device 40, PTD 20, andoptionally electromagnetic field generator 62 of image analysis system50. In FIG. 5A, patient 10 inhales and patient 10 is scanned usingimaging device 40 which generates a population of images 402 of theanatomy of patient 10 and markers 22 at inspiration. As shown, patient10 may place their arms above their head as they inhale, and this may beconsidered a total lung capacity (TLC) scan. In FIG. 5B, patient 10exhales and patient 10 is scanned using imaging device 40 whichgenerates a population of images 404 of the anatomy of patient 10 andmarkers 22 at expiration. As shown, patient 10 may place their armsbelow their head, and this may be considered a functional residualcapacity (FRC) scan. The Functional Residual Capacity is the lung volumeat the end of a normal expiration, when the muscles of respiration arecompletely relaxed. At FRC (and typically at FRC only), the tendency ofthe lungs to collapse is exactly balanced by the tendency of the chestwall to expand. In various embodiments, the population of images 402,404 may be two-dimensional (2D) images. In other embodiments, forexample, the population of images 402, 404 may be three-dimensional (3D)images. Additionally, the population of images 402, 404 may berespiratory gated by tracking the location of localization elements 24of PTD 20 by image analysis system 50 and/or navigation system 70 usingEM field generator 62, 82 during image generation. In other embodiments,for example, the population of images 402, 404 may be gated using anytype of device known for generating a physiological signal for gating.

In various embodiments, for example, instead of patient 10 holding aninspiration or expiration state, a cine loop of images may be generatedin conjunction with the patient's respiratory cycle information from PTD20. Thus the cine loop comprises a population of images generated frominspiration to expiration where the population of images are gated tothe respiratory cycle of patient 10 using PTD 20. This can serve tolimit registration point selection, in order to be consistent with thepatient's respiratory cycle that a 3D dataset such as CT, MR, or PET hasacquired. This technique advantageously maximizes registration accuracy,a major flaw in conventional systems in the prior art.

As described above, imaging device 40 is in data communication withimage analysis system 50 and/or navigation system 70 and sends,transfers, copies and/or provides the population of images 402, 404taken during the first time interval associated with patient 10 to imageanalysis system 50 and/or navigation system 70. As shown in FIG. 5C,image analysis system 50 and/or navigation system 70 compiles thepopulation of images 402 at inspiration into a 3D image data subset 406of the anatomy of patient 10 and markers 22 at inspiration (referred toherein as inspiration 3D image data subset 406). Additionally, imageanalysis system 50 and/or navigation system 70 compiles the populationof images 404 at expiration into a 3D image data subset 408 of theanatomy of patient 10 at expiration (referred to herein as expiration 3Dimage data subset 408). The inspiration 3D image data subset 406 and theexpiration 3D image data subset 408 are then stored in an image dataset400 in memory component 54, 74 of image analysis system 50 and/ornavigation system 70.

Additionally, acquiring a population of images at both inspiration andexpiration may assist navigation of a steerable catheter during a secondtime interval. Referring now to FIGS. 6A-6C, in addition to segmentingthe markers 22 of PTD 20 from the population of images 402, 404generated during the first time interval, processor 52 of image analysisworkstation 50 generates three-dimensional models of the airway ofpatient 10 by segmenting the 3D image data subsets 406, 408. In variousembodiments, segmentation of the airway may be accomplished using aniterative region growing technique wherein a seed voxel in the airway isselected as an initialization parameter. Voxels neighboring the seedvoxel are then evaluated to determine whether they are a part of theairway, form the wall surrounding the airway, or form other tissue.Following segmentation, a surface mesh of the airway may be generated toproduce a surface skeleton. The surface of the airway may then berendered.

As shown in FIG. 6A, a three-dimensional model of the airway of patient10 at inspiration (“inspiration 3D airway model 410”) is generated bysegmenting the inspiration 3D image data subset 406. FIG. 6A shows anInspiration/arms-up pathway registration; this is, generally speaking,the preferred image scan acquisition state for automatic segmentation ofthe tracheo-bronchial tree. Processor 52 may also segment one or moretarget tissues 420 (e.g., lesions, lymph nodes, blood vessels, tumors,etc.) which may be navigated to during a second time interval using avariety of medical devices as described more fully elsewhere herein. Thesegmentation of the target tissue(s) 420 may be refined to definedifferent characteristics of the target tissue, such as, for example,density of the target tissue. Additional image data formats may also beloaded into processor 52, such as, for example, PET or MR and processor52 may be able to map the CT, PET, and/or MR data to one another.

As shown at FIG. 6B, a three-dimensional model of the airway of patient10 at expiration (“expiration 3D airway model 412”) is generated bysegmenting the expiration 3D image data subset 408. As discussed above,a variety of segmentation algorithms known in the art may be used togenerate the inspiration and expiration 3D airway models 410, 412. FIG.6B shows, in contrast to FIG. 6A, an FRC/arms-down segmentation. Becausethe patient's 10 lungs are more full of air at inspiration than atexpiration, the inspiration 3D airway model 410 includes more structurethan the expiration 3D airway model 412. Accordingly, as shown in FIG.6B, expiration 3D airway model 412 includes fewer structure(s) and thestructure(s) are in different locations and/or orientations than atinspiration. However during a procedure such as directing a navigatedsteerable catheter to a target tissue within the airway of patient 10(e.g., during a second time interval), the breathing cycle of patient 10may be closer to tidal breathing. That is, patient 10 usually neverreaches full inspiration during the procedure and thus if thesegmentation of the airways of patient 10 at inspiration is used fornavigation purposes, there will be significant error in the registrationof the segmented airway to patient 10.

In certain embodiments, a hybrid “Inspiration-Expiration” 3D airwaymodel 414 is constructed as shown in FIG. 6C using the inspiration 3Dairway model 410 and the expiration 3D airway model 412. The hybrid“Inspiration-Expiration” 3D airway model 414 may be used to reduce oreliminate the errors in registration. To construct the hybrid“Inspiration-Expiration” 3D airway model 414, a population ofdeformation vector fields is calculated by processor 52, 72 of imageanalysis system 50 and/or navigation system 70. The deformation vectorfield comprises vectors from one or more voxels in the inspiration 3Dairway model 410 to one or more corresponding voxels in the expiration3D airway model 412. After the deformation vector field is calculated,the inspiration 3D airway model 410 is deformed to the expiration stateof patient 10 using the deformation vector field. Accordingly, thevoxels in the inspiration 3D airway model 410 are deformed to match thelocation, shape, and orientation of the airways of patient 10 atexpiration. This results in the hybrid “Inspiration-Expiration” 3Dairway model 414, wherein the hybrid “Inspiration-Expiration” 3D airwaymodel 414 contains all of the structural information of the airways ofpatient 10 depicted in inspiration 3D airway model 410. However, thisstructural information is now more closely matched to the location,shape, and orientation of the airways of patient 10 depicted inexpiration 3D airway model 412. Accordingly, the deformation vectorsrepresent not only a change in location of the structure of the airwaybut a change in shape of the structure of the airway from inspiration toexpiration.

FIG. 7 , illustrates a 3D representation of hybrid“Inspiration-Expiration” 3D airway model 414 which includes a targettissue 420 segmented by processor 52, 72. This 3D representation ofhybrid “Inspiration-Expiration” 3D airway model 414 may include surfaceinformation. Hybrid “Inspiration-Expiration” 3D airway model 414 mayadditionally include navigation pathway 416. Image analysis system 50and/or navigation system 70 may calculate navigation pathway 416 fromthe entry of the airway to the location of target tissue 420. In certainembodiments, navigation pathway 416 may be an optimal endobronchial pathto a target tissue. For example, navigation pathway 416 may representthe closest distance and/or closest angle to the target tissue. Aphysician or other healthcare professional may follow navigation pathway416 during an image guided intervention to reach the location of targettissue 420.

Although target tissue 420 locations and navigation pathway(s) 416 maybe automatically calculated by image analysis system 50 and/ornavigation system 70, a physician or other healthcare professional maymanually adjust target tissue 420 locations and/or navigation pathway(s)416.

In general, the embodiments described herein have applicability in“Inspiration to Expiration”-type CT scan fusion. According to variousmethods, the user navigates on the expiration 3D image data subset 408for optimal accuracy, while using the inspiration 3D image data subset406 to obtain maximum airway segmentation. In one embodiment, forexample, a user could complete planning and pathway segmentation on theinspiration 3D image data subset 406 of patient 10. Preferably, adeformation vector field is created between at least two datasets (e.g.,from inspiration 3D image data subset 406 to expiration 3D image datasubset 408). The deformation or vector field may then be applied to thesegmented vessels and/or airways and navigation pathway 416 and targettissue 420 locations. In these and other embodiments, the deformation orvector field may also be applied to multiple image datasets or in aprogressive way to create a moving underlying image dataset that matchesthe respiratory or cardiac motion of patient 10.

By way of example, in certain embodiments, “Inspiration to Expiration”CT fusion using the lung lobe centroid and vector change to modify anairway model may also be applicable. In accordance with variousembodiments, this technique is used to translate and scale each airwaybased on the lung lobe change between inspiration images and expirationimages. The lung is constructed of multiple lobes and these lobes arecommonly analyzed for volume, shape, and translation change. Each lobechanges in a very different way during the patient's breathing cycle.Using this information to scale and translate the airways that arelocated in each lobe, it is possible to adapt for airway movement. Thisscaled airway model may then be linked to the 4D tracking of the patientas described herein.

In various aspects, the systems and methods described herein involvemodifying inspiration images generated by imaging device 40 (e.g., CT,CT/PET, MRI, etc.) to the expiration cycle for navigation. It is wellunderstood that the patient's airways are contained within multiplelobes of the lung. It is also understood that airways significantlychange between inspiration and expiration. In certain embodiments, toincrease the accuracy of the map for navigation, it may be beneficial toinclude the detail of the inspiration images, coupled with the abilityto navigate it accurately during expiration. For many patients, theexpiration state may be the most repeatable point in a patient's breathcycle. In preferred embodiments, this modification may be carried out inaccordance with the following steps:

1) Generate a population of images of patient 10 at both inspiration andexpiration using imaging device 40;

2) Segment the airways in both the inspiration and expiration images;

3) Segment the lung lobes in both the inspiration and expiration images(as the lung lobes are identifiable in both the inspiration andexpiration images with a high degree of accuracy);

4) Determine a volume difference for each lung lobe between inspirationand expiration, use this change to shrink the airway size from theinspiration to the expiration cycle. Preferably, this is done for eachindividual lobe, as the percentage change will typically be differentfor each lobe.

5) Determine the centroid for each lung lobe and the vector change inmotion from the main carina in both inspiration images and expirationimages. This vector may then be used to shift the airways that areassociated with each lung lobe. A centroid for the airway may becalculated based on the segmented branches. For each airway branch inthe segmentation, it includes a tag that associates it with therespective lung lobe. The central airway including the main carina andinitial airway branches for each lobe that is linked according to theexpiration scan location of these points. Next, a plane may be definedusing the main carina and initial airway branch exits to determine thevector change for each lobe.

Among the lobes to modify, for example:

left inferior lobe—the bottom lobe of the lung on the left side ofpatient 10;

left superior lobe—the top lobe of the lung on the left side of patient10.

right inferior lobe—the bottom lobe of the lung on the right side ofpatient 10;

right middle lobe—the middle lobe of the lung on the right side ofpatient 10;

right superior lobe—the top lobe of the lung on the right side ofpatient 10.

Exemplary calculations are as follows:

Inspiration Airway—Left Inferior Lobe (LIL)×70% (reduction in volumeInspiration to Expiration calculated)=ExAirwayLIL;

Determine Expiration Central Airway points (Main Carina and InitialAirway branches) based upon segmentation;

Shift ExAirwayLIL by vector distance (3 cm, 45 degrees up and back frommain carina) that LIL centroid moved from inspiration to expiration.

Preferably, this process is repeated for each lobe. In certainembodiments, the completion of 5 lobes will result in a hybrid“Inspiration-Expiration” 3D airway model for patient 10.

In various embodiments, the target location for the patient may beselected in the expiration images and applied to the hybrid“Inspiration-Expiration” 3D airway model 414. Alternatively, it may beselected in the inspiration images and adjusted based on the same orsimilar criteria as the inspiration airways. In either case, it may beadjusted individually or linked to the airway via a 3D network and movedin the same transformation.

A deformation field may also be included in the analysis in variousother embodiments described herein. For example, the deformation fieldmay be applied to fuse 3D fluoroscopic images to CT images to compensatefor different patient orientations, patient position, respiration,deformation induced by the catheter or other instrument, and/or otherchanges or perturbations that occur due to therapy delivery or resectionor ablation of tissue.

Following the generation of hybrid “Inspiration-Expiration” 3D airwaymodel 414, during a second time interval, a medical procedure is thenperformed on patient 10 with PTD 20 coupled to patient 10 at the samelocation as during the first time interval when the population ofpre-procedural images were taken. Preferably, the second time intervalimmediately follows the first time interval. However, in certainembodiments, second time interval may occur several hours, days, weeksor months after the first time interval. After hybrid“Inspiration-Expiration” 3D airway model 414 is generated and one ormore target tissues 420 are identified and one or more navigationpathways 416 are calculated, this information is transferred from imageanalysis system 50 to navigation system 70. This transfer may be doneaccording to the DICOM (Digital Imaging and Communications in Medicine)standard as known in the art. It will be understood that the transfermay be done using any method and according to any standard withoutdeparting from the scope of the invention. For example, this transfermay be accomplished between image analysis system 50 to navigationsystem 70 using a variety of methods, including, but not limited to, awired connection, a wireless connection, via CD, via a USB device, etc.

It should be noted that image dataset 400 may be supplemented, replacedor fused with an additional image dataset. In one embodiment, forexample, during the second time interval an additional population ofimages may be taken. In other embodiments, for example, after the secondtime interval an additional population of images may be taken. Bygenerating one or more additional image datasets, potential changedphysical parameters of patient such as patient 10 movement, anatomicalchanges due to resection, ablation, general anesthesia, pneumothorax,and/or other organ shift may be accounted for during the procedure.Accordingly, images from CT-Fluoro, fluoroscopic, ultrasound or 3Dfluoroscopy may be imported into image analysis system 50 and/ornavigation system 70.

Using the respiratory signal derived from PTD 20, navigation system 70selects an image from the population of pre-procedural images 402, 404taken during the first time interval that indicates a distance or isgrouped in a similar sequence of motion between corresponding markers 22at a given instant in time, that most closely approximates or matchesthe distance or similar sequence of motion between the selectedlocalization elements 24. The process of comparing the distances isdescribed in more detail below. Thus, navigation system 70 displaysimages corresponding to the actual movement of the targeted anatomyduring the medical procedure being performed during the second timeinterval. The images illustrate the orientation and shape of thetargeted anatomy during a path of motion of the anatomy, for example,during inhaling and exhaling.

FIG. 8 illustrates an example set of distances or vectors d1 through d6between a set of markers 22, labeled m1 through m9 that are disposed atspaced locations on PTD 20. As described above, a population ofpre-procedural images is taken of a patient 10 to which PTD 20 iscoupled during a first time interval. The distances between markers 22are determined for multiple instants in time through the path of motionof the dynamic body (e.g., the respiratory cycle of the patient). Then,during a medical procedure, performed during a second time interval,localization elements 24 (not shown in FIG. 8 ) proximate the locationof markers 22 provide position data for localization elements 24 tolocalization device 76 (not shown in FIG. 8 ). Navigation system 70 usesthe position data to determine distances or vectors between localizationelements 24 for multiple instants in time during the medical procedureor second time interval.

FIG. 9A shows an example of distance or vector data from localizationdevice 76. Vectors al through a6 represent distance data for one instantin time and vectors n1 through n6 for another instant in time, during atime interval from a to n. As previously described, the vector data maybe used to select an image from the population of pre-procedural imagesthat includes distances between the markers m1 through m9 thatcorrespond to or closely approximate the distances al through a6 fortime a, for example, between the localization elements. The same processmay be performed for the vectors n1 through n6 captured during time n.

One method of selecting the appropriate image from the population ofpre-procedural images 402, 404 is to execute an algorithm that sums allof the distances al through a6 and then search for and match this sum toan image containing a sum of all of the distances d1 through d6 obtainedpre-procedurally from the image data that is equal to the sum of thedistances al through a6. When the difference between these sums is equalto zero, the relative position and orientation of the anatomy or dynamicbody D during the medical procedure will substantially match theposition and orientation of the anatomy in the particular image. Theimage associated with distances d1 through d6 that match or closelyapproximate the distances al through a6 may then be selected anddisplayed. For example, FIG. 9B illustrates examples of pre-proceduralimages, Image a and Image n, of a dynamic body D that correspond to thedistances al through a6 and n1 through n6, respectively. An example ofan algorithm for determining a match is as follows:

Does Σa_(i)=Σd_(i) (i=1 to 6 in this example) OR

Does Σ(a_(i)−d_(i))=0 (i=1 to 6 in this example).

If yes to either of these, then the image is a match to the vector ordistance data obtained during the medical procedure.

FIG. 10 is a flowchart illustrating a method according to an embodimentof the invention. A method 100 includes at step 102 generating imagedata during a pre-procedural or first time interval. As discussed above,a population of images are generated of a dynamic body, such as patient10, using imaging device 40 (e.g., CT Scan, MRI, etc.). The image datais associated with one or more images generated of PTD 20 coupled to adynamic body, where PTD 20 includes two or more markers 22. In otherwords, the image data of the dynamic body is correlated with image datarelated to PTD 20. The one or more images may be generated using avariety of different imaging devices as described previously. The imagedata include an indication of a position of a first marker and anindication of a position of a second marker, as illustrated at step 104.The image data include position data for multiple positions of themarkers during a range or path of motion of the dynamic body over aselected time interval. As described above, the image data includeposition data associated with multiple markers, however, only two aredescribed here for simplicity. A distance between the position of thefirst marker and the position of the second marker is determined formultiple instants in time during the first time interval, at step 106.As also described above, the determination may include determining thedistance based on the observable distance between the markers on a givenimage. The image data, including all of the images received during thefirst time interval, the position, and the distance data is recorded ina memory component at step 108.

Then at step 110, during a second time interval, while performing amedical procedure on patient 10 with PTD 20 positioned on patient 10 atsubstantially the same location, position data is received for a firstlocalization element and a second localization element. Localizationelements 24 of PTD 20 are proximate markers 22, such that the positiondata associated with localization elements 24 is used to determine therelative position of markers 22 in real-time during the medicalprocedure. The position data of localization elements 24 are recorded ina memory component at step 112.

A distance between the first and second localization elements isdetermined at step 114. Although only two localization elements 24 aredescribed, as with the markers, position data associated with more thantwo localization elements may be received and the distances between theadditional localization elements may be determined.

The next step is to determine which image from the population of imagestaken during the first time interval represents the relative positionand/or orientation of the dynamic body at a given instant in time duringthe second time interval or during the medical procedure. To determinethis, at step 116, the distance between the positions of the first andsecond localization elements at a given instant in time during thesecond time interval determined in step 114 are compared to thedistance(s) determined in step 106 between the positions of the firstand second markers obtained with the image data during the first timeinterval.

An image is selected from the first time interval that best representsthe same position and orientation of the dynamic body at a given instantin time during the medical procedure. To do this, the difference betweenthe distance between a given pair of localization elements during thesecond time interval is used to select the image that contains the samedistance between the same given pair of markers from the image datareceived during the first time interval. This is accomplished, forexample, by executing an algorithm to perform the calculations. Whenthere are multiple pairs of markers and localization elements, thealgorithm may sum the distances between all of the selected pairs ofelements for a given instant in time during the second time interval andsum the distances between all of the associated selected pairs ofmarkers for each instant in time during the first time interval when thepre-procedural image data was received.

When an image is found that provides the sum of distances for theselected pairs of markers that is substantially the same as the sum ofthe distances between the localization elements during the second timeinterval, then that image is selected at step 118. The selected image isthen displayed at step 120. The physician or other healthcareprofessional may then observe the image during the medical procedure.Thus, during the medical procedure, the above process may becontinuously executed such that multiple images are displayed and imagescorresponding to real-time positions of the dynamic body may be viewed.

In addition to tracking the location of PTD 20, navigation system 70(see FIG. 3 ) may also track any type of device which includes one ormore localization elements. The localization elements in the medicaldevices may be substantially similar or identical to localizationelements 24 of PTD 20. The devices preferably include medical devices,including, but not limited to, steerable catheters, needles, stents,ablation probes, biopsy devices, guide wires, forceps devices, brushes,stylets, pointer probes, radioactive seeds, implants, endoscopes, energydelivery devices, therapy delivery devices, delivery of energy activatedsubstances (e.g., porfimer sodium) and energy devices, radiofrequency(RF) energy devices, cryotherapy devices, laser devices, microwavedevices, diffuse infrared laser devices, etc. In certain embodiments,the location of these devices are tracked in relation to PTD 20. Inother embodiments, for example, these devices are tracked in relation toelectromagnetic field generator 62, 82. It is also envisioned that atleast some of these medical devices may be wireless or have wirelesscommunications links. It is also envisioned that the medical devices mayencompass medical devices which are used for exploratory purposes,testing purposes or other types of medical procedures.

One embodiment of a medical device which may be tracked by navigationsystem 70 is illustrated in FIGS. 11 and 11A. In one embodiment of thepresent invention, a navigated surgical catheter that is steerable 600(referred herein to as “steerable catheter”) may be used to gain accessto, manipulate, remove, sample or otherwise treat tissue within the bodyincluding, but not limited to, for example, heart or lung tissue.Steerable catheter 600 comprises an elongate flexible shaft 602 having aproximal end portion 604, a distal end portion 606 terminating in tip607, and one or more working channels 608 extending from proximal endportion 604 to tip 607. As shown in FIG. 11A, one or more localizationelements 610 that are detectable by navigation system 70 are disposedproximate the distal end portion 606 of elongate flexible shaft 602.Accordingly, the position and orientation (POSE) of localizationelements 610 are tracked by localization device 76 of navigation system70. The one or more localization elements 610 are connected by wire 611to navigation system 70; in alternative embodiments, the one or morelocalization elements 610 may be wirelessly connected to navigationsystem 70. In certain embodiments, localization elements 610 comprisesix (6) degree of freedom (6DOF) electromagnetic coil sensors. In otherembodiments, localization elements 610 comprise five (5) degree offreedom (5DOF) electromagnetic coil sensors. In other embodiments,localization elements 610 comprise other localization devices such asradiopaque markers that are visible via fluoroscopic imaging andechogenic patterns that are visible via ultrasonic imaging. In yet otherembodiments, localization elements 610 may be, for example, infraredlight emitting diodes, and/or optical passive reflective markers.Localization elements 610 may also be, or be integrated with, one ormore fiber optic localization (FDL) devices. Accordingly, in certainembodiments, localization elements 610 may be substantially similar oridentical to localization elements 24 of PTD 20. In other embodimentsthe steerable catheter may be non-navigated, such that it does notinclude any localization elements.

Steerable catheter 600 further comprises handle 612 attached to theproximal end portion 604 of elongate flexible shaft 602. Handle 612 ofsteerable catheter 600 includes steering actuator 614 wherein distal endportion 606 is moved “up” and “down” relative to proximal end portion604 by manipulating steering actuator 614 “up” and “down,” respectively.Additionally, distal end portion 606 is moved “left” and “right”relative to proximal end portion 604 by rotating handle 612 “left” and“right,” respectively, about handle longitudinal axis 613. It will beunderstood that steering actuator 614 and handle 612 are connected to asteering mechanism (not shown) on the inside of steerable catheter 600which is connected to distal end portion 606 of elongate flexible shaft602 for causing the deflection in distal end portion 606. Port 616,disposed on handle 612, provides access to working channel(s) 608 inelongate flexible shaft 602 of steerable catheter 600, such that amedical device may be inserted into working channel(s) 608 through port616.

As shown in FIGS. 12A and 12B, any number of medical devices ortherapies may be inserted into working channel(s) 608 and/or extendedout of tip 607 to deliver the medical devices or therapies to a targettissue. The medical devices may include, but are not limited to, imagingdevices 633, tissue sensing devices 632, biopsy devices, therapydevices, steerable catheters, endoscopes, bronchoscopes, percutaneousdevices, percutaneous needles, pointer probes, implants, stents, guidewires, stylets, etc. In certain embodiments, imaging devices 633include, but are not limited to, bronchoscopic video cameras 630,endobronchial ultrasound (EBUS) devices 634, optical coherencetomography (OCT) devices, probe based Confocal Laser Endomicroscopy(pCLE) devices, or any known imaging device insertable into workingchannel 608 of steerable catheter 600. Tissue sensing device 632 may beany type of device which may be used to determine the presence of atarget tissue in patient 10. In certain embodiments, tissue sensingdevice 632 may include, but is not limited to, imaging device 633, acell analysis device, a cancer detecting device, an exhaled breathcondensate analyzer, a physiological characteristic sensor, a chemicalanalysis device, an aromatic hydrocarbon detection device, vacuumcollection device, etc. The sensitivity of certain of the tissuesampling devices, such as aromatic hydrocarbon detection devices aredependent upon the density of the sample collected. Thus, by navigatingsteerable catheter 600 near the desired target tissue a sample of higherdensity may be captured and analyzed. Additionally, a vacuum collectiondevice may be navigated using steerable catheter 600 to near the desiredtarget tissue and/or an airway branch within one or two segments of thedesired target tissue, and an air sample may be captured. In certainembodiments, therapy devices include, but are not limited to, ablationprobes, energy delivery devices, radioactive seeds, delivery of energyactivated substances (e.g., porfimer sodium) and energy devices,radiofrequency (RF) energy devices, cryotherapy devices, laser devices,microwave devices, diffuse infrared laser devices, fluids, drugs,combinations thereof, or the like). In certain embodiments, biopsydevices include, but are not limited to, needles, forceps devices,brushes, etc. In certain embodiments, steerable catheter 600 may alsoinclude a suction capability.

As illustrated in FIG. 12A, for example, in certain embodiments, imagingdevice 633 is a bronchoscopic video camera 630. Bronchoscopic videocamera 630 may be inserted into working channel 608 and/or extended outdistal end portion 606 of navigated steerable catheter 600. By insertingbronchoscopic video camera 630 into working channel 608 of steerablecatheter 600, steerable catheter 600 may be used like a typicalsteerable bronchoscope, as described more fully elsewhere herein.

As shown in FIG. 12B, tissue sensing device 632 may be an imaging device633, wherein imaging device 633 is an endobronchial ultrasound (EBUS)device 634; however, as described above, it will be understood thatimaging device 633 may include, but is not limited to, bronchoscopicvideo camera 630, an optical coherence tomography (OCT) device, a probebased Confocal Laser Endomicroscopy (pCLE) device, or any known imagingdevice insertable into working channel 608 of steerable catheter 600.

In embodiments, where tissue sensing device 632 is imaging device 633,imaging device 633 may be able to generate a population of images of thetarget tissue(s), wherein the target tissue(s) may be in the airway, onthe wall of the airway, in the wall of the airway, and/or beyond thewall of the airway. That is, the imaging device(s) may be able togenerate images of target tissue(s) not only inside the airway, but maygenerate images of target tissue(s) and other anatomical structuresinside and/or beyond the wall of the airway. Thus in certainembodiments, sub-surface target tissue may be imaged using the imagingdevice(s). Accordingly, using endobronchial ultrasound (EBUS) device634, an optical coherence tomography (OCT) device, a probe basedConfocal Laser Endomicroscopy (pCLE) device, etc. while tracking theposition and orientation (POSE) of localization element 610 of steerablecatheter 600, as described herein, multiple 3D volumes of image dataregarding target tissue(s) and other anatomical structures inside and/orbeyond the wall of the airway may be collected and a larger 3D volume ofcollected data may be constructed. Knowing the 3D location andorientation of the multiple 3D volumes will allow the physician or otherhealthcare professional to view a more robust image of, for example,pre-cancerous changes of target tissue(s) in patient 10. This data mayalso be correlated to pre-acquired or intra-procedurally acquired imagedataset 400 to provide additional information.

Additionally, in certain embodiments wherein steerable catheter 600includes multiple working channels 608, multiple medical devices may beinserted into the multiple working channels 608. For example,bronchoscopic video camera 630 may be inserted into one working channeland a medical device such as a needle, forceps device or a brush may beinserted into a second working channel. Accordingly, a real-time imagefeed from bronchoscopic video camera 630 may be used to view theoperation of the medical device. Although a steerable catheter has beendescribed, it will be understood that any type of steerable medicaldevice may be used in accordance with the methods described herein,including, but not limited to, endoscopes, bronchoscopes, etc. withoutdeparting from the scope of the invention.

Another embodiment of a medical device which may be tracked bynavigation system 70 is illustrated in FIGS. 13 and 13A. In oneembodiment of the present invention, a percutaneous needle 650 may beused to gain access to, manipulate, remove, sample or otherwise treattarget tissue within patient 10 including, but not limited to, forexample, target tissue on and/or in the heart or lung. Percutaneousneedle 650 comprises an elongate shaft or cannula 652 having a proximalend portion 654, a distal end portion 656 terminating in tip 657, andone or more working channels 658 may extend from proximal end portion654 to tip 657. Percutaneous needle 650 further includes handle 652attached to the proximal end portion 654. Port 667, disposed on handle652, provides access to working channel(s) 658 in cannula 652 ofpercutaneous needle 650, such that a medical device may be inserted intoworking channel(s) 658 through port 667. Any number of medical devicesor therapies, as described herein, may be inserted into workingchannel(s) 658 and/or extended out of tip 657 to deliver the medicaldevices or therapies (e.g., steerable catheters, needles, stents,ablation probes, biopsy devices, guide wires, forceps devices, brushes,stylets, pointer probes, radioactive seeds, implants, endoscopes, energydelivery devices, therapy delivery devices, delivery of energy activatedsubstances (e.g., porfimer sodium) and energy devices, radiofrequency(RF) energy devices, cryotherapy devices, laser devices, microwavedevices, diffuse infrared laser devices, fluids, drugs, combinationsthereof, or the like) to a target tissue.

As shown in FIG. 13A, one or more localization elements 660 that aredetectable by navigation system 70 are disposed proximate the distal endportion 656 of cannula 652. Accordingly, the position and orientation(POSE) of localization elements 660 are tracked by localization device76 of navigation system 70. The one or more localization elements 660are connected by wire 661 to navigation system 70; in alternativeembodiments, the one or more localization elements 660 may be wirelesslyconnected to navigation system 70.

In certain embodiments, localization elements 660 comprise six (6)degree of freedom (6DOF) electromagnetic coil sensors. In otherembodiments, localization elements 660 comprise five (5) degree offreedom (5DOF) electromagnetic coil sensors. In other embodiments,localization elements 660 comprise other localization devices such asradiopaque markers that are visible via fluoroscopic imaging andechogenic patterns that are visible via ultrasonic imaging. In yet otherembodiments, localization elements 660 may be, for example, infraredlight emitting diodes, and/or optical passive reflective markers.Localization elements 660 may also be, or be integrated with, one ormore fiber optic localization (FDL) devices. Accordingly, in certainembodiments, localization elements 660 may be substantially similar oridentical to localization elements 24 of PTD 20 and/or localizationelements 610 of steerable catheter 600.

While localization element 660 is illustrated proximate distal endportion 656, it will be understood that localization element 660 may belocated in other locations of percutaneous needle 650 without departingfrom the scope of the invention. For example, in certain embodiments,localization element 660 may be disposed proximate the proximal endportion 654 and/or proximate handle 662. Navigation system 70 may beable to determine the location of tip 657 in relation to the location ofPTD 20 by knowing the location of localization element 660 in relationto tip 657. For example, if localization element 660 is disposed athandle 662, navigation system 770 may be able to determine the locationof tip 657 in relation to the position of localization element 660 ifthe length between tip 657 and localization element 660 is input intonavigation system 70.

In other embodiments, percutaneous needle 650 is non-navigated, suchthat it does not include any localization elements. However, thelocation of percutaneous needle 650 may still be tracked by navigationsystem 70 if a medical device containing a localization element isinserted into working channel 658 of percutaneous needle 650.

In various embodiments, any of the medical devices described herein thatmay be inserted into working channel(s) 608, 658 of steerable catheter600 and/or percutaneous needle 650 may be tracked individually with anintegrated localization element (e.g., an electromagnetic (EM) sensor).Accordingly, the medical devices may be tip tracked. Additionally,wherein the inserted medical device is an ablation probe, ablationmodels may be displayed to assist in optimal placement of the ablationprobe for treatment. It will be understood that the medical devices maybe delivered endobronchially, percutaneously, and/or endobronchially andpercutaneously simultaneously.

Referring again to navigation system 70, navigation system 70 maydisplay on display 80 multiple images which may assist a physician orother healthcare professional in conducting the methods describedherein. Image dataset 400 generated during the first time interval maybe registered to patient 10 using PTD 20. As described above,localization elements 24 of PTD 20 are proximate markers 22 and becauseone or more markers 22 of PTD 20 are visible in image dataset 400 andlocalization elements 24 corresponding to the one or more markers 22 aretracked by navigation system 70, image dataset 400 may be registered topatient 10. This registration may be manually accomplished or may beautomatically accomplished by navigation system 70.

In addition to or alternative to registration using PTD 20, registrationmay be completed by different known techniques. First, point-to-pointregistration may be accomplished by identifying points in an image spaceand then touching the same points in patient space. These points aregenerally anatomical landmarks that are easily identifiable on thepatient. Second, lumen registration may be accomplished by generating apoint cloud within the airways of patient 10 and matching the shape ofthe generated point cloud to an inspiration 3D airway model 410, anexpiration 3D airway model 412, and/or a hybrid “Inspiration-Expiration”3D airway model 414. Using four-dimensional tracking (4D) the pointcloud may be generated at the appropriate respiration cycle to matchinspiration 3D airway model 410, an expiration 3D airway model 412,and/or a hybrid “Inspiration-Expiration” 3D airway model 414. Generationof a point cloud is more fully described in U.S. Ser. No. 13/773,984,entitled “Systems, Methods and Devices for Forming Respiratory-GatedPoint Cloud for Four Dimensional Soft Tissue Navigation,” filed on Feb.22, 2013, which is hereby incorporated by reference. Third, surfaceregistration may involve the generation of a surface in patient 10 spaceby either selecting multiple points or scanning, and then accepting thebest fit to that surface in image space by iteratively calculating withprocessor 72 until a surface match is identified. Fourth, repeatfixation devices entail repeatedly removing and replacing a device(i.e., dynamic reference frame, etc.) in known relation to patient 10 orimage fiducials of patient 10. Fifth, two-dimensional (2D) imagedatasets may be registered to three-dimensional (3D) image datasetswherein, the two dimensional image datasets may include, but are notlimited to, fluoroscopic images, ultrasound images, etc. and thethree-dimensional (3D) image datasets may include, but are not limited,to computed tomography (CT) images, fused computed tomography-positronemission tomography (CT/PET) images, magnetic resonance imaging (MRI)images. Sixth, automatic registration may be accomplished by firstattaching a dynamic reference frame to patient 10 prior to acquiringimage data. It is envisioned that other known registration proceduresare also within the scope of the present invention, such as thatdisclosed in U.S. Pat. No. 6,470,207, entitled Navigational Guidance viaComputer-Assisted Fluoroscopic Imaging”, filed on Mar. 23, 1999, whichis hereby incorporated by reference.

After image dataset 400 is registered to patient 10, navigation system70 displays on display 80 a variety of images as illustrated in FIG. 14. For example, as shown in panel 700, hybrid “Inspiration-Expiration” 3Dairway model 414 may be displayed. Additionally, as shown in panel 700,an indicia 718 (shown as a crosshair) of the location of steerablecatheter 600 is displayed. In certain embodiments, for example, indicia718 indicates the location of distal end portion 606 of steerablecatheter 600. In other embodiments, for example, indicia 718 indicatesthe location of localization element 610 of steerable catheter 600. Inyet other embodiments, for example, indicia 718 indicates the locationof tip 607 of steerable catheter 600. That is, navigation system 70 maybe able to display an indicia indicating the location of a portion ofsteerable catheter 600 based on the tracked location of localizationelement 610. For example, if localization element 610 is disposed 5 mmfrom tip 607 of steerable catheter 600, the 5 mm distance may be takeninto account by navigation system 70 and the indicia of tip 607indicating the location of tip 607 may be displayed and not the locationof localization element 610. An indicia 720 (shown as a circle) of aninitial target tissue location may also be displayed on display 80 bynavigation system 70 as shown in panel 700. Indicia 718, 720 are shownas a crosshair and circle, respectively; however it is envisioned thatother indicia may be used to indicate the location of steerable catheter600, initial target tissue location, confirmed target tissue location,location of percutaneous needle 650, and/or any other target tissue ormedical device. For example, indicia may have different shapes, colors,sizes, line weights and/or styles, etc. without departing from the scopeof the invention.

Furthermore, navigation system 70 may be able to simulate and displayaxial, coronal and oblique images based on the position and orientation(POSE) of localization element 610 of steerable catheter 600, as shownin panels 702, 704, and 706. To simulate these views, navigation system70 may modify one or more images from image dataset 400 using knownimage manipulation techniques. Additionally, navigation system 70 maysimulate and/or display orthogonal image slices, oblique or off-axisimage slices, volume rendered images, segmented images, fused modalityimages, maximum intensity projection (MIPS) images, video, and videoenhanced images. As shown, indicia of 718 of steerable catheter 600and/or an indicia 720 of an initial target tissue location may also bedisplayed, as shown in panels 702, 704, and 706.

In various embodiments as shown in panel 712, navigation system 70 alsosimulates a virtual volumetric scene within the body of patient 10, suchas the airways of patient 10, from a point of view of a medical device,such as steerable catheter 600, as it is being navigated into and/orthrough patient 10. This virtual volumetric scene is acomputer-generated visualization of a bronchoscopy procedure andsimulates what would be viewed by a bronchoscopic video camera insertedinto the airways. To simulate the virtual volumetric scene, navigationsystem 70 modifies one or more images from image dataset 400 using knownimage manipulation techniques. For example, navigation system 70 may beable to simulate the virtual volumetric scene using inspiration 3Dairway model 410, expiration 3D airway model 412, and/or hybrid“Inspiration-Expiration” 3D airway model 414. Accordingly navigationsystem 70 renders an internal view of 3D airway model(s) 410, 412,and/or 414 based on a virtual bronchoscope video camera position, forexample, by applying certain surface properties (e.g., Lambertian),diffuse shading model(s), and perspective projection camera model(s).Virtual lighting and shading may be applied to the rendered view tofurther enhance the virtual volumetric scene. The field of view (FOV)may be changed to match the field of view of bronchoscopic video camera630 (see FIG. 12A). The point of view may be adjusted to matchbronchoscopic video camera 630 or to display a virtual volumetric scenefrom different points along the airway or outside the airway. Navigationsystem 70 may also be able to display a navigation pathway 416 in thevirtual volumetric scene. Accordingly, the virtual volumetric scene mayallow a physician or other healthcare professional to review thenavigation pathway 416 prior to inserting steerable catheter 600 and/orother medical device into patient 10. Additionally, in certainembodiments, an indicia of the location of localization element 610 ofsteerable catheter 600 and/or an indicia of an initial target tissuelocation may also be displayed.

Additionally, in various embodiments as shown in panel 716, navigationsystem 70 also displays a real-time image feed from bronchoscopic videocamera 630 inserted into working channel 608 of steerable catheter 600.The real-time image feed may be static images or moving video. Thereal-time image feed may assist the physician or other healthcareprofessional in navigating steerable catheter 600 to proximate theinitial location of the target tissue. Thus by inserting bronchoscopicvideo camera 630 into working channel 608 of steerable catheter 600 (seeFIG. 12A), steerable catheter 600 may be used like a typical steerablebronchoscope. Typical steerable bronchoscopes are used to visuallyinspect the airways of a patient and have a fixed bronchoscopic videocamera in addition to one or more working channels. Typical steerablebronchoscopes may have steering actuators and steering mechanisms thatpermit them to be steered much like steerable catheter 600. Because thebronchoscopic video camera of a typical steerable bronchoscope is fixedduring manufacture of the steerable bronchoscope, the “up” orientationof the image feed from the bronchoscopic video camera as displayed tothe physician or other healthcare professional is aligned with the “up”direction of the steering actuator of the typical steerablebronchoscope. That is, the orientation of the real-time image feed fromthe typical steerable bronchoscope is registered to the orientation ofthe steering directions of the typical steerable bronchoscope.Accordingly, when the physician or other healthcare professional steersthe typical steerable bronchoscope “up,” the image feed will move “up.”Additionally, steering the typical steerable bronchoscope “down,”“left,” and “right,” the image feed will move “down,” “left,” and“right,” respectively.

However, because the bronchoscopic video camera is fixed (i.e.,non-removable) in the typical steerable bronchoscope, the outsidediameter of the typical steerable bronchoscope must be large enough toalso accommodate one or more working channels. Due to the large outsidediameter of typical steerable bronchoscopes, certain portions of theairways may be unreachable by the steerable catheter because thediameter of the airway may be too small. Accordingly, it may bedesirable to use steerable catheter 600 which may have a smaller outsidediameter than the typical steerable bronchoscope. Bronchoscopic videocamera 630 may be inserted working channel 608 of steerable catheter 600and a real-time image feed is displayed to the physician or otherhealthcare professional. Using the real-time image feed, the physicianor other healthcare professional may navigate steerable catheter 600 tovery small diameter portions of the airway which were previouslyinaccessible by a typical steerable bronchoscope. Once the physician orother healthcare professional has reached the desired target tissue withsteerable catheter 600, the physician or other healthcare professionalmay remove bronchoscopic video camera 630 from working channel 608 ofsteerable catheter 600 and insert one or more other medical devices intoworking channel 608 as described more fully elsewhere herein.Additionally, because bronchoscopic video camera 630 is not fixed insteerable catheter 600, the ratio(s) of the diameter(s) of workingchannel(s) 608 of steerable catheter 600 to the outside diameter ofsteerable catheter 600 may be much higher than the ratio(s) of thediameter(s) of working channel(s) of a typical steerable bronchoscope tothe outside diameter of a typical steerable bronchoscope.

While the removable nature of bronchoscopic video camera 630 providesthe above mentioned benefits, because the bronchoscopic video camera 630is not fixed in steerable catheter 600, the “up” orientation of theimage feed from bronchoscopic video camera 630 as displayed to thephysician or other healthcare professional may not be aligned with the“up” direction of steering actuator 614 of steerable catheter 600. Thatis, depending on how the physician or other healthcare professionalinserts bronchoscopic video camera 630 into working channel 608 ofsteerable catheter, what appears as “up” in the real-time image feedfrom bronchoscopic video camera 630 may not correspond to an “up”steering input to steering actuator 614. Accordingly, the real timeimage feed may be rotated relative to the expected steering direction.This may introduce uncertainty and or confusion to the physician orother healthcare professional. For example, the physician or otherhealthcare professional may see an airway on the left hand side of thedisplayed real-time image feed and may accordingly manipulate handle 612of steerable catheter to cause distal end portion 606 of steerablecatheter 600 to steer left. However, because the orientation of thereal-time image feed is not aligned with the with steering actuator 614of steerable catheter 600, the airway that the physician or otherhealthcare professional thought was on the left hand side of thereal-time image feed is not actually reachable by a left hand steeringinput to steerable catheter 600. Accordingly, the orientation of theimage feed from bronchoscopic video camera 630 as displayed to thephysician or other healthcare professional may not be aligned withsteering actuator 614 of steerable catheter 600. Thus, to ensure thatthe physician or other healthcare professional is navigating down thedesired airway, the “up” orientation of the real-time image feed frombronchoscopic video camera 630 as displayed to the physician or otherhealthcare professional should be aligned with the “up” direction ofsteering actuator 614 of steerable catheter 600.

Referring now to FIG. 15 , one method of registering the real-time imagefeed from a bronchoscopic video camera 630 to a steerable catheter 600is described. At step 900, bronchoscopic video camera 630 is insertedinto working channel 608 of steerable catheter 600. In certainembodiments, tip 631 (see FIG. 12A) of bronchoscopic video camera 630 ispositioned proximate or extended past tip 607 of steerable catheter 600.At step 902, a real-time image feed of one or more reference points isgenerated using bronchoscopic video camera 630, wherein the orientationof the reference point(s) is known. That is, the physician or otherhealthcare professional may know or ascertain the orientation of thereference point(s) independently from the real-time image feed. At step904, the physician or other healthcare professional orients handle 612of steerable catheter 600 to a neutral position. Preferably, handle 612of steerable catheter 600 is considered to be in a neutral position whenlongitudinal axis 613 of handle 612 is substantially vertical, when no“up” or “down” steering input is applied to steerable catheter 600 bysteering actuator 614, and when no “left” or “right” steering input isapplied to steerable catheter 600 by rotation of handle 612 aboutlongitudinal axis 613. When in the neutral position, it is not requiredthat elongate flexible shaft 602 of steerable catheter 600 be straight.Elongate flexible shaft 602 may be flexed; however it is contemplatedthat no additional steering inputs are applied to steerable catheter600.

At step 906, the real-time image feed of bronchoscopic video camera 630is displayed on display 80 of navigation system 70. At step 908, thereal-time image feed is registered to steerable catheter 600 by rotatingthe displayed real-time image feed so that the reference point(s) in thereal-time image feed are matched to the known orientation of thereference point(s). In certain embodiments, the physician or healthcareprofessional manually rotates the real-time image feed on display 80 ofnavigation system 70 using user interface device 84 (e.g., keyboard,mouse). In other embodiments, for example, navigation system 70 mayautomatically rotate the real-time image feed on display 80 ofnavigation system 70.

Optionally, the registration may be confirmed by steering steerablecatheter 600 to cause a deflection of distal end portion 606 of elongateflexible shaft 602 in a direction and observing that the displayedreal-time image feed moves in that same direction. For example, ifphysician or other healthcare professional manipulates steering actuator614 to cause an “up” deflection in distal end portion 606 of elongateflexible shaft 602, the displayed real-time image feed will also move“up.” Similarly, if the physician or other healthcare professionalmanipulates steering actuator 614 to cause a “down” deflection in distalend portion 606 of elongate flexible shaft 602, the displayed real-timeimage feed will also move “down.” Additionally, if the physician orother healthcare professional rotates handle 612 “left” or “right” tocause a “left” or “right” deflection in distal end portion 606 ofelongate flexible shaft 602, the displayed real-time image feed willalso move “left” or “right.” Accordingly, after the real-time image feedis registered to steerable catheter 600, what is displayed as “up,”“down,” “left,” and/or “right,” corresponds to “up,” “down,” “left,” and“right” steering inputs to steerable catheter 600. That is, theorientation of the displayed real-time image feed is matched to thesteering mechanics of steerable catheter 600.

As shown in FIGS. 16 and 16A-16C, a jig 802 may be used in conjunctionwith the method of registering the real-time image feed from abronchoscopic video camera 630 to a steerable catheter 600 described inFIG. 15 . As shown in FIG. 16 , jig 802 may include receiver 803 intowhich distal end portion 606 of steerable catheter 600 may be placed.Jig 802 further includes three round objects 804 which serve as thereference points described above. Accordingly, when viewed along arrowA, round objects 804 are known to be oriented as shown in FIG. 16A. Whenplaced in jig 802, bronchoscopic video camera 630 is inserted intoworking channel 608 of steerable catheter 600 and handle 612 ofsteerable catheter 600 is oriented in a neutral position, as describedabove. Thus if the three round objects 804 are rotated a certain anglein the displayed real-time image feed from bronchoscopic video camera630 as shown in FIG. 16B, the real-time image feed needs to beregistered by rotating the displayed real-time image feed so that thethree round objects 804 in the real-time image feed are matched to theknown orientation of the three round objects 804 as shown in FIG. 16C.While reference points are illustrated as three round objects 804, itwill be understood that any shape object may be used as a referencepoint, including, but not limited to, a T-shaped object, a cross-shapedobject, a square shaped object, etc. Additionally, while three referencepoints are illustrated, it will be understood that jig 802 may includeone or more reference points. In other embodiments, for example, jig 802may include a picture or pattern which serves as the one or morereference points.

Another embodiment of the method of registering the real-time image feedfrom a bronchoscopic video camera 630 to a steerable catheter 600 isshown in FIG. 17 . At step 910, bronchoscopic video camera 630 isinserted into working channel 608 of steerable catheter 600. In certainembodiments, tip 631 (see FIG. 12A) of bronchoscopic video camera 630 ispositioned proximate or extended past tip 607 of steerable catheter 600.At step 912, steerable catheter 600 is inserted into the airway ofpatient 10. At step 914, a real-time image feed of one or more referencepoints is generated using bronchoscopic video camera 630, the referencepoint(s) comprising anatomical feature(s) of the airway wherein theorientation of the anatomical feature(s) is known. In certainembodiments, the anatomical feature(s) may include the right mainbronchus (RMB) and the left main bronchus (LMB).

As shown in FIG. 18A, it is generally understood that the RMB and theLMB of most patients are oriented at about a 3 o'clock position andabout a 9 o'clock position respectively when viewed with a typicalsteerable bronchoscope. Referring again to FIG. 17 , at step 916, thephysician or other healthcare professional orients handle 612 ofsteerable catheter 600 to a neutral position. Preferably, handle 612 ofsteerable catheter 600 is considered to be in a neutral position whenlongitudinal axis 613 of handle 612 is substantially vertical, when no“up” or “down” steering input is applied to steerable catheter 600 bysteering actuator 614, and when no “left” or “right” steering input isapplied to steerable catheter 600 by rotation of handle 612 aboutlongitudinal axis 613. When in the neutral position, it is not requiredthat elongate flexible shaft 602 of steerable catheter 600 be straight.Elongate flexible shaft 602 may be flexed; however it is contemplatedthat no additional steering inputs are applied to steerable catheter600.

At step 918, the real-time image feed of bronchoscopic video camera 630is displayed on display 80 of navigation system 70. As shown in FIG.18B, the displayed real-time image feed of bronchoscopic video camera630 shows the RMB and LMB rotated such that the RMB appears at about a 2o'clock position and the LMB appears at about an 8 o'clock position. Atstep 920, the real-time image feed is registered to steerable catheter600 by rotating the displayed real-time image feed so that theanatomical feature(s) in the real-time image feed are matched to theknown orientation of the anatomical feature(s). Thus as shown in FIG.18C, after registration, the displayed real-time image feed ofbronchoscopic video camera 630 shows the RMB and LMB at about a 3o'clock position and at about a 9 o'clock position, respectively. Incertain embodiments, the physician or healthcare professional manuallyrotates the real-time image feed on display 80 of navigation system 70using user interface device 84 (e.g., mouse). In other embodiments, forexample, navigation system 70 may automatically rotate the real-timeimage feed on display 80 of navigation system 70. The method mayoptionally continue according to steps illustrated in FIG. 17A asdescribed more fully elsewhere herein.

Optionally, the registration may be confirmed by steering steerablecatheter 600 to cause a deflection of distal end portion 606 of elongateflexible shaft 602 in a direction and observing that the displayedreal-time image feed moves in that same direction. For example, ifphysician or other healthcare professional manipulates steering actuator614 to cause an “up” deflection in distal end portion 606 of elongateflexible shaft 602, the displayed real-time image feed will also move“up.” Similarly, if the physician or other healthcare professionalmanipulates steering actuator 614 to cause a “down” deflection in distalend portion 606 of elongate flexible shaft 602, the displayed real-timeimage feed will also move “down.” Additionally, if the physician orother healthcare professional rotates handle 612 “left” or “right” tocause a “left” or “right” deflection in distal end portion 606 ofelongate flexible shaft 602, the displayed real-time image feed willalso move “left” or “right.” Accordingly, after the real-time image feedis registered to steerable catheter 600, what is displayed as “up,”“down,” “left,” and “right,” corresponds to “up,” “down,” “left,” and“right” steering inputs to steerable catheter 600.

In some embodiments, the registration of the real-time image feed tosteerable catheter 600 may be enhanced by correcting image distortion inthe real time image feed. For example, bronchoscopic video camerastypically include fish-eye lenses which increase the field of view ofthe bronchoscopic video camera thus providing the physician or otherhealthcare professional with a larger view of the airway of patient 10.However, the fish-eye lenses introduce barrel distortion into thereal-time image feed. Due to this barrel distortion, the interpretationof the real-time image feed may be compromised. Correcting for thisimage distortion in the real-time image feed provides a more accuratedepiction of the airway of patient 10, thus permitting an enhancedregistration of the real-time image feed to steerable catheter 600.

Referring again to FIG. 14 , in yet other embodiments, the virtualvolumetric scene displayed in panel 712 may be registered to thereal-time image feed from a bronchoscopic video camera 630 displayed inpanel 716. However, as steerable catheter 600 is navigated through theairways of patient 10, steerable catheter 600 may be positioned in sucha way such that what appears “up” in the real-time image feed may notcorrespond to the physical “up” direction of patient 10. That is, thephysical “up” of patient 10 usually corresponds to the anteriordirection of patient 10 as patient 10 is oriented during the procedure.Typically, patient 10 is in the supine position and thus, the physical“up” of the patient will correspond to an actual “up.” However, incertain situations, patient 10 may be in different orientations duringthe procedure, such as on their side or chest However, the virtualvolumetric scene displayed in panel 712 is shown with the chest ofpatient 10 facing up. Accordingly, the real-time image feed as shown inpanel 716 may not match the virtual volumetric scene displayed in panel712. To assist the physician or other healthcare professional innavigating down the correct airway, the virtual volumetric scene may beregistered to the real-time image feed, wherein the real-time image feedhas been registered to steerable catheter 600.

In some embodiments, image correction is applied to the real-time imagefeed to assist in registering the virtual volumetric scene to thereal-time image feed. To register the virtual volumetric scene as shownin panel 712 with the real-time image feed from bronchoscopic videocamera 630 as shown in panel 716, the lens distortion of the real-timeimage feed must be corrected or the same lens distortion must be appliedto the virtual volumetric scene.

After correcting the real-time image feed for lens distortion, virtualvolumetric scene is registered to the real-time image feed. An initialregistration may be performed in a region of the airway that is easilylocatable with steerable catheter 600, such as the trachea for example.Thus the virtual volumetric scene as shown in panel 712 may be manuallyor automatically rotated to match one or more airway structure(s) (e.g.,RMB and LMB) visible in both the virtual volumetric scene and thereal-time image. In various embodiments, a matching algorithm may thenbe used to maintain registration of the virtual volumetric scene to thereal-time image feed as steerable catheter 600 is navigated through theairway. Other registration methods known in the art may also be appliedwithout departing from the scope of the invention. For example, thevirtual volumetric scene may be registered to the real-time image feedusing intensity based maximization of information mutual to thereal-time image feed and the virtual volumetric scene instead ofmatching structures. In other embodiments, for example, surface normalsof the real-time image feed may be calculated using a linear shape fromshading algorithm based on the unique camera and/or lightingconfigurations of bronchoscopic video camera 630. The virtual volumetricscene may then be registered to the real-time image feed by matching thecalculated surface normal with surface normal of the virtual volumetricscene. Accordingly, the registration of the virtual volumetric scene tothe real-time image feed may cause both the real-time image feed and thevirtual volumetric scene to be displayed on display 80 with “up” as“up.”

In yet other embodiments, the registration of the virtual volumetricscene to the real-time image feed may be enhanced by registering thereal-time image feed to localization element 610 of steerable catheter600. By registering the real-time image feed to localization element610, both the real-time image feed and/or the virtual volumetric scenemay be shown in the “up” orientation on display 80 no matter what theposition and orientation (POSE) of localization element 610 in steerablecatheter 600 is as tracked by navigation system 70. The physician orother healthcare professional may always expect that an “up” steeringinput on steering actuator 614 will always result in the displayedreal-time image moving “up.” Thus, even if physician or other healthcareprofessional moves handle 612 of steerable catheter such thatlongitudinal axis 613 is not substantially vertical and thereby causes arotation of distal end portion 606 of steerable catheter 600, becausethe real-time image feed is registered to steerable catheter 600 and tolocalization element 610, navigation system 70 may display real-timeimage feed and/or virtual volumetric scene with “up” as “up.”Accordingly, the physician or other healthcare professional may still beable to easily determine how to manipulate steering actuator 614 ofsteerable catheter 600 to navigate steerable catheter 600 alongnavigation pathway 416 displayed in panel 712.

Referring now to FIG. 17A, a method of registering the real-time imagefeed from bronchoscopic video camera 630 to localization element 610 ofsteerable catheter 600 is described. Preferably, registration of thereal-time image feed from bronchoscopic video camera 630 to localizationelement 610 of steerable catheter 600 is performed after the real-timeimage feed from bronchoscopic video camera 630 is registered tosteerable catheter 600. At step 922, navigation system 70 tracks thelocation of localization element 610 of steerable catheter 600. At step924, the orientation of the registered real-time image feed with respectto localization element 610 is determined. Referring now to FIG. 19 , asection view of steerable catheter 600 is shown to aid in describing theregistration of the real-time image feed from bronchoscopic video camera630 to localization element 610 of steerable catheter 600. For purposesof simplicity not all structure of steerable catheter 600, localizationelement 610, and bronchoscopic video camera 630 are illustrated. Asshown in FIG. 19 , O_(i) represents the un-registered orientation of thereal-time image feed from bronchoscopic video camera 630. O_(R)represents the orientation of the real-time image feed frombronchoscopic video camera 630 after the real-time image feed frombronchoscopic video camera 630 is registered to steerable catheter 600.Thus during registration of the real-time image feed from bronchoscopicvideo camera 630 to steerable catheter 600, the real-time image feed wasrotated by angle Θ (see FIGS. 15, 17 steps 908, 920 respectively).

Thus, referring again to FIG. 17A, determining the orientation of theregistered real-time image feed with respect to localization element 610at step 924, comprises determining the angle β from O_(R) to the trackedlocation of localization element 610. At step 926, the determinedorientation (e.g., angle β) is recorded to navigation system 70.Accordingly, after the real-time image feed from bronchoscopic videocamera 630 is registered to localization element 610 of steerablecatheter 600, the real-time image feed and/or the virtual volumetricscene may be shown in the “up” orientation on display 80 regardless ofthe position and orientation (POSE) of localization element 610 insteerable catheter 600 as tracked by navigation system 70. Additionally,by registering the real-time image feed from bronchoscopic video camera630 to localization element 610 of steerable catheter 600, theregistration of the virtual volumetric scene may be maintained to thereal-time image feed as steerable catheter 600 is navigated in theairway of patient 10.

In various embodiments as described above, registering the real-timeimage feed from a bronchoscopic video camera 630 to a steerable catheter600 permits displaying one or more navigational aids over the real-timeimage feed from bronchoscopic video camera 630, wherein the navigationalaids are registered to the real-time image feed. In certain embodiments,the navigational aids may be determined using the hybrid“Inspiration-Expiration” 3D airway model 414. For example, in certainembodiments, navigation system 70 may overlay navigation pathway 416onto the real-time image feed from bronchoscopic video camera 630. Inother embodiments, for example, navigation system may also overlaydirectional cues such as arrows or other indicators on the real-timeimage feed from bronchoscopic video camera 630. Integrating navigationalaids, including but not limited to navigation pathway 416 and/ordirectional cues, with the real-time image feed may assist the physicianor other healthcare professional in navigating steerable catheter 600 tothe desired target tissue. Accordingly, in certain embodiments whereinnavigational aids are overlaid onto real-time image feed, a virtualvolumetric scene does not need to be displayed on display 80 ofnavigation system 70.

Although registering the real-time image feed from a bronchoscopic videocamera 630 to a steerable catheter 600 has been described in detailherein, it will be understood that image feeds from other imagingdevices 633 inserted into working channel 608 of steerable catheter 600may be registered in similar manners. The imaging devices 633 mayinclude, but are not limited to, endobronchial ultrasound (EBUS) device634 (see FIG. 12B), an optical coherence tomography device (OCT), andprobe based Confocal Laser Endomicroscopy (pCLE).

Returning to FIG. 14 , navigation system 70 may also display a graphicalrepresentation 708 of the respiratory cycle of patient 10 monitoredusing PTD 20. In certain embodiments, one or more of the images and/orindicia displayed in panels 700, 702, 704, 706, 712 and 716 aredisplayed as a function of the monitored respiratory state. That is,images in image dataset 400 and/or generated from image dataset 400 aredisplayed on display 80 that depict the anatomy of patient 10 at themonitored respiratory state. For example, when the patient is atexpiration as monitored by PTD 20, images of the anatomy of the patientdepicting the anatomy at expiration are displayed. Accordingly, when thepatient is at inspiration as monitored by PTD 20, images of the anatomyof patient 10 depicting the anatomy at inspiration are displayed. Inother embodiments, one or more of the images displayed in panels 700,702, 704, 706, 712 and 716 may not be displayed as a function of themonitored respiratory state. That is, images in image dataset 400 and/orgenerated from image dataset 400 are displayed on display 80 that depictthe anatomy of patient 10 at one respiratory state. For example, whenthe patient is at expiration and inspiration as monitored by PTD 20,images of the anatomy of patient 10 depicting the anatomy at expirationare displayed. In embodiments where images are not displayed accordingto the monitored respiratory state, an indication 710 of whether thedisplayed images match the monitored respiratory state may be shown(e.g., “Respiration Matched”, “Respiration out of Sync”).

Additionally, the display of indicia of the locations of the targettissue and/or indicia of the location of various medical devices may besynchronized or gated with an anatomical function, such as the cardiacor respiratory cycle, of patient 10. That is, in certain embodiments,the indicia are displayed on display 80 as a function of the monitoredrespiratory state. In certain instances, the cardiac or respiratorycycle of patient 10 may cause the indicia to flutter or jitter withinpatient 10. In these instances, the indicia will likewise flutter orjitter on the image(s) displayed on display 80.

To eliminate the flutter of the indicia on the displayed image(s), theposition and orientation (POSE) of localization elements 610, 660 isacquired at a repetitive point within each cycle of either the cardiaccycle or the respiratory cycle of patient 10. To synchronize theacquisition of position data for localization elements 610, 660,navigation system 70 may use a timing signal (e.g., respiratory phasesignal) generated by PTD 20; however one skilled in the art will readilyrecognize other techniques for deriving a timing signal that correlateto at least one of the cardiac or respiratory cycle or other anatomicalcycle of the patient.

As described above, the indicia indicate the location of steerablecatheter 600 and percutaneous needle 650 based on the location oflocalization elements 610, 660 tracked by navigation system 70 assteerable catheter 600 and percutaneous needle 650 are navigated by thephysician or other healthcare profession on and/or within patient 10.Rather than display the indicia on a real-time basis, the display of theindicia may be periodically updated based on the timing signal from PTD20. In various embodiments, PTD 20 may be connected to navigation system70. Navigation system 70 may then track localization elements 610, 660in response to a timing signal received from PTD 20. The position of theindicia may then be updated on display 80. It is readily understood thatother techniques for synchronizing the display of an indicia based onthe timing signal are within the scope of the present invention, therebyeliminating any flutter or jitter which may appear on the displayedimage(s). It is also envisioned that a path (or projected path) ofsteerable catheter 600, percutaneous needle 650, and/or other medicaldevices may also be illustrated on the displayed image(s).

Utilizing the devices, systems, and/or methods described herein, amethod of endobronchially confirming the location of a target in thelung of a patient and percutaneously intercepting the target at theconfirmed location may be performed. In various embodiments, this methodis performed during a second time interval after an image dataset 400 isgenerated during a first time interval. As illustrated in FIGS. 20A-20B,an embodiment of a method of endobronchially confirming the location ofa target is illustrated. At step 1000, PTD 20 is affixed to the externalsurface of a patient 10. At step 1002, the respiratory state of patient10 may be monitored by tracking the location of PTD 20 using navigationsystem 70. At step 1004, navigation system 70 displays an image fromimage dataset 400 on display 80 as a function of the monitoredrespiratory state. The displayed image is selected from one or moreimages in image dataset 400 and/or is generated by navigation system 70using one or more images in image dataset 400. The displayed image isregistered to PTD 20. At step 1006, an initial location of one or moretarget tissues in image dataset 400 is determined. This initial locationof the target tissue is where it is believed that a target tissue islocated within patient 10.

In certain embodiments, for example, the initial location of the targettissue(s) is determined after image dataset 400 is generated during thefirst time interval. In certain embodiments, for example, the initiallocation of the target tissue(s) may be selected at the start of and/orduring the second time interval. The initial location of the targettissue(s) may be determined automatically using nodule detection orsegmentation algorithms carried out by processor 52 of image analysissystem 50 and/or processor 72 of navigation system 70. Additionally oralternatively, a physician or other healthcare professional manuallyidentifies a target tissue on an image displayed on display 60 of imageanalysis system 50 and/or display 80 of navigation system 70. Thephysician or other healthcare professional may then determine theinitial location of the target tissue(s) by selecting the target tissuedepicted on display(s) 50, 80 using user interface device(s) 64, 84(e.g., by clicking on displayed target tissue with a mouse) or someother point selection tool. In other embodiments, the initial locationof the target tissue may be determined by the physician or otherhealthcare professional using nodule segmentation tools and/or usingnodule density information. An indicia 720 of the initial target tissuelocation may then be displayed on display 80 as shown in FIG. 14 .

Returning to FIG. 20A, at step 1008, a physician or other healthcareprofessional navigates steerable catheter 600 through the airway ofpatient 10 to a position proximate the initial location of the targettissue. Additionally, in certain embodiments, an imaging device 633 suchas bronchoscopic video camera 630 (see FIG. 12A) is inserted intoworking channel 608, navigation system 70 displays on display 80 thereal-time image feed of the inside of the airway of patient 10 generatedby bronchoscopic video camera 630 as shown in panel 716 of FIG. 14 . Asdescribed above the real-time image feed may be registered to steerablecatheter 600. In certain embodiments, navigation system 70 may overlaynavigation pathway 416 onto the real-time image feed from bronchoscopicvideo camera 630. In other embodiments, for example, navigation systemmay also overlay directional cues such as arrows or other indicators onthe real-time image feed from bronchoscopic video camera 630.Integrating navigational aids, including but not limited to navigationpathway 416 and/or directional cues, with the real-time image feed mayassist the physician or other healthcare professional in navigatingsteerable catheter 600 to the desired target tissue. Accordingly, incertain embodiments wherein navigational aids are overlaid ontoreal-time image feed, a virtual volumetric scene does not need to bedisplayed on display 80 of navigation system 70.

With reference again to FIG. 20A, as steerable catheter 600 is navigatedthrough the airway of patient 10, at step 1010, navigation system 70tracks the location of localization element 610 of steerable catheter60. As described above, an indicia of 718 of the location of steerablecatheter 600 may also be displayed on display 80 as shown in panels 700,702, 704, and 706 of FIG. 14 .

Referring now to FIG. 20B, the method continues at step 1012, whereinformation regarding the presence of the target tissue is generatedusing tissue sensing device 632 inserted into working channel 608 ofsteerable catheter 600. In certain embodiments, tissue sensing device632 may be imaging device 633 inserted into the airway of patient 10,such as, for example, endobronchial ultrasound (EBUS) device 634 (seeFIG. 12B), an optical coherence tomography device (OCT), and/or probebased Confocal Laser Endomicroscopy (pCLE). Imaging device 633 may beextended out tip 607 of steerable catheter 600 and may generate apopulation of images of the target tissue. Where imaging device 637 isEBUS device 634, EBUS device 634 may be a radial EBUS device or a linearEBUS device. Illustrated in FIG. 21 is an exemplary image 721 of thetarget tissue generated by a radial EBUS device which may be displayedon display 80. In other embodiments, tissue sensing device 632 mayinclude, but is not limited to, a cell analysis device, a cancerdetecting device, an exhaled breath condensate analyzer, a physiologicalcharacteristic sensor, a chemical analysis device, an aromatichydrocarbon detection device, etc.

Returning to FIG. 20B, a confirmed location of the target is using thegenerated information regarding the presence of the target tissue andthe tracked location of localization element 610 of steerable catheter600. For example, if tissue sensing device 632 is an imaging device 633which generates a population of images of the target tissue, navigationsystem 70 tracks the extension (x), if any, of imaging device 633 inrelation to localization element 610. By tracking the extension (x) inrelation to localization element 610 and the position and orientation(POSE) of localization element 610, navigation system 70 knows thecoordinates at which the population of images of the target tissue aregenerated and may thus determine the actual location and size of thetarget tissue within patient 10. In certain embodiments, the confirmedlocation of the target tissue is determined in relation to the locationof PTD 20. In other embodiments, for example, the confirmed location ofthe target tissue is determined in relation to the location ofelectromagnetic (EM) field generator 82 of navigation system 70.

At step 1016, after the confirmed location of the target tissue isdetermined, the confirmed location of the target tissue is recorded. Inone embodiment, for example, recording the confirmed location of thetarget tissue comprises recording a three-dimensional (3D) location ofthe confirmed target tissue in relation to PTD 20. In anotherembodiment, for example, recording the confirmed location of the targettissue comprises recording a three-dimensional (3D) location of theconfirmed target tissue in relation to electromagnetic (EM) fieldgenerator 82 of navigation system 70. In one embodiment, for example,recording the confirmed location of the target tissue comprisesrecording four-dimensional data (4D) comprising a three-dimensional (3D)location of the confirmed target tissue in relation to PTD 20 and therespiratory state of patient 10 at the time the location of the targettissue was confirmed. In another embodiment, for example, recording theconfirmed location of the target tissue comprises recordingfour-dimensional data (4D) comprising a three-dimensional (3D) locationof the confirmed target tissue in relation to electromagnetic (EM) fieldgenerator 82 of navigation system 70 and the respiratory state ofpatient 10 at the time the location of the target tissue was confirmed.In yet another embodiment, for example, recording the confirmed locationof the target tissue comprises recording four-dimensional (4D) datacomprising a three-dimensional location (3D) of the confirmed targettissue in relation to PTD 20 and a cardiac state of the patient at thetime the location of the target tissue was confirmed. In yet anotherembodiment, for example, recording the confirmed location of the targettissue comprises recording four-dimensional (4D) data comprising athree-dimensional location (3D) of the confirmed target tissue inrelation to electromagnetic (EM) field generator 82 and a cardiac stateof the patient at the time the location of the target tissue wasconfirmed. In various embodiments, this confirmed location of the targettissue may then be applied to one or more images from image dataset 400depicting the airway at the respiratory state of patient 10 at the timethe location of the target tissue was confirmed. This information isrecorded in memory component 74 of navigation system 70.

At step 1018, the confirmed location of the target tissue is displayedon display 80 of navigation system 70 in one or more images from imagedataset 400. In certain embodiments, the displayed image(s) depict theairway of the patient at the respiratory state of patient 10 at the timethe location of the target tissue was confirmed. As shown in FIG. 22 ,navigation system 70 may display an indicia 722 (shown as crosshairbounded by a square) of the confirmed location of the target tissue in avariety of images, including but not limited to, hybrid“Inspiration-Expiration” 3D airway model 414 in panel 700, axial,coronal and oblique images in panels 702, 704 (shown enlarged in FIG.22A), and 706, respectively, and virtual volumetric scene in panel 712.Navigation system 70 may be able to display an indicia 720 (shown ascircle in crosshair bounded by a circle) of the initial location of thetarget tissue, an indicia 722 of the confirmed location of the targettissue, and an indicia 718 (shown as crosshair) of steerable catheter600. The method may optionally continue according to steps illustratedin FIGS. 20C and 20E as described more fully elsewhere herein.

After the confirmed location of the target tissue is recorded, thephysician or other healthcare professional can return to the confirmedlocation of the target tissue using a medical device, such as steerablecatheter 600 or percutaneous needle 650, without needing to re-registerthe patient. Accordingly, because, in certain embodiments, the confirmedlocation of the target tissue is recorded in relation to the location ofpatient tracking device 20, the physician or other healthcareprofessional can navigate medical device to the confirmed location ofthe target tissue knowing the location of patient tracking device 20.For example, in certain embodiments, the physician or other healthcareprofessional navigates to the confirmed location of the target tissuewherein navigation system 70 displays on display 80 only an indicia 722of the confirmed location of the target tissue, an indicia 718 ofsteerable catheter 600, and/or an indicia 734 of percutaneous needle 650(see FIG. 23 ). Using one or more of indicia 722, 718, 734, physician orother healthcare professional navigates medical device to the confirmedlocation of the target tissue without needing navigation system 70 todisplay hybrid “Inspiration-Expiration” 3D airway model 414, one or moreimages from image dataset 400, navigation pathway 416, and/or real timeimage feed from bronchoscopic video camera 630.

Additionally, because, in certain embodiments, the confirmed location ofthe target tissue is recorded in relation to the location ofelectromagnetic (EM) field generator 82, the physician or otherhealthcare professional can navigate medical device to the confirmedlocation of the target tissue if patient 10 has not moved relative tolocalization device 76. For example, in certain embodiments, thephysician or other healthcare professional navigates to the confirmedlocation of the target tissue wherein navigation system 70 displays ondisplay 80 an indicia 722 of the confirmed location of the targettissue, an indicia 718 of steerable catheter 600, and/or an indicia 734of percutaneous needle 650. Using one or more of indicia 722, 718, 734,physician or other healthcare professional navigates medical device tothe confirmed location of the target tissue without needing navigationsystem 70 to display hybrid “Inspiration-Expiration” 3D airway model,one or more images from image dataset 400, navigation pathway 416,and/or real time image feed from bronchoscopic video camera 630.

Due to a variety of factors including, but not limited to, registrationerrors, shifting of patient location, changes in patient anatomy, theinitial target location determined at step 1006 may not match the actualconfirmed location of the target determined in step 1014. Accordingly,without performing this confirmation step, a biopsy or medical therapydelivered to the initial target location may only partially interceptthe actual target tissue or may be performed at an incorrect locationsuch as healthy tissue. Insufficient and/or incorrect sampling ortreatment of the target tissue and/or healthy tissue may lead tomisdiagnoses and/or reduced treatment efficacy. Thus, by confirming theactual location of the target tissue in relation to PTD 20 and/orelectromagnetic (EM) field generator 82, intercepting (e.g., sampling,treating) the target tissue may be more accurately carried out in avariety of ways. Consequently, a physician or other healthcareprofessional may have a higher confidence that they are intercepting thetarget tissue. In certain embodiments, for example, the target tissuemay be sampled using a variety of medical devices including, but notlimited to, forceps devices, needles, brushes, etc. Treatment may alsobe endobronchially delivered to the confirmed location of the targettissue using a variety of medical devices including, but not limited to,ablation probes, radioactive seeds, implants, energy delivery devices,therapy delivery devices, delivery of energy activated substances (e.g.,porfimer sodium) and energy devices, radiofrequency (RF) energy devices,cryotherapy devices, laser devices, microwave devices, diffuse infraredlaser devices, fluids, drugs, combinations thereof, or the like.

In certain embodiments, the target tissue may not be reachable usingendobronchial methods, accordingly, after the location of the targettissue is endobronchially confirmed, the target tissue may bepercutaneously intercepted. The percutaneous interception may be carriedout using a percutaneous device. Percutaneous device may preferably bepercutaneous needle 650 described above. Because percutaneous needle 650includes localization element 660, the position and orientation (POSE)of tip 657 is tracked by navigation system 70. Accordingly, navigationsystem 70 calculates and displays a trajectory of percutaneous needle650 based on where percutaneous needle 650 is located and oriented byphysician or other healthcare professional. However, in variousembodiments, for example, percutaneous device may include, but is notlimited to percutaneous needle 650, a thoracic wedge resection device, abiopsy gun, a tracked core biopsy device, and/or any other medicaldevice which may be used to percutaneously intercept a target tissue.The percutaneous devices preferably include a localization element sothat the position and orientation (POSE) of the percutaneous devices maybe tracked by navigation system 70.

Referring now to FIG. 20C, at step 1020, navigation system 70 displayson display 80 one or more trajectories from an entry point on thesurface of patient 10 to the confirmed location of the target tissue. Incertain embodiments, a displayed trajectory may be a suggestedtrajectory calculated by navigation system 70 wherein the suggestedtrajectory is the shortest distance from the confirmed location of thetarget tissue to the external surface of patient 10. Navigation system70 may utilize procedural position of patient 10 such as supine, prone,or laying on the left or right side to calculate the suggestedtrajectory. Accordingly, navigation system 70 may display a suggestedtrajectory that does not require altering the procedural position ofpatient 10. For example, if the trajectory having the shortest distancefrom the confirmed location of the target tissue to the external surfaceof patient 10 requires entry through the back of patient 10, but patient10 is supine, an alternative suggested trajectory may be calculated anddisplayed which permits entry through the chest or side of patient 10.

In certain embodiments, the suggested trajectory may be calculated thatextends through the longest axis or longest diameter of the targettissue to ensure that the amount of target tissue sampled and/or treatedis increased and/or maximized. Additionally, the patient 10 specificsegmented target tissue may also have characteristics such as highdensity or spiculations that identify preferred regions to sample and/ortreat. For example, in certain embodiments, the suggested trajectory maybe calculated to extend through spiculations of the target tissue. Inother embodiments, for example, a change in size of the target tissuemay be seen between inspiration and expiration scans. In certainsituations, this apparent change in size may be the result of infectedtissue near the target tissue changing in size from inspiration toexpiration. Typically, however, the target tissue will not change insize from inspiration to expiration, accordingly image analysis system50 and/or navigation system 70 may be able to identify the target tissuebased on a minimal or no change in density, size, location, and shapefrom inspiration to expiration. The suggested trajectory may thus becalculated to extend through such portions of the target tissue.

Additionally or alternatively, in certain embodiments, a displayedtrajectory may be an actual trajectory calculated by navigation system70 wherein the actual trajectory is the based on where percutaneousneedle 650 is located and oriented by physician or other healthcareprofessional. Accordingly, in certain embodiments, navigation system 70may be able to display on display 80 both a suggested trajectory and anactual trajectory of percutaneous needle 650. Thus, a physician or otherhealthcare professional may move tip 657 of percutaneous needle 650along the body of patient 10 and may orient percutaneous needle 650 sothat the suggested trajectory and the actual trajectory displayed bynavigation system 70 on display 80 are in alignment. Once the actualtrajectory and the suggested trajectory are in alignment, the physicianor other healthcare professional inserts percutaneous needle 650 intopatient along the actual trajectory. In other embodiments, for example,no suggested trajectory may be displayed.

FIG. 23 illustrates one embodiment where navigation system 70 displayson display 80 suggested and actual trajectories from an entry point onthe surface of patient 10 to the confirmed location of the targettissue. Panels 724 and 726 illustrate views that navigation system 70may display. The displayed images may be selected from one or moreimages in image dataset 400 or may be generated by navigation system 70using one or more images in image dataset 400. Additionally, indicia 722(shown as crosshair bounded by a square) of the confirmed location ofthe target tissue and suggested trajectory 730 from entry point 732 tothe confirmed location of the target tissue are displayed on display 80.Furthermore, an indicia 734 of the location of percutaneous needle 650is displayed. In certain embodiments, for example, indicia 734 indicatesthe location of distal end portion 656 of percutaneous needle 650. Inother embodiments, for example, indicia 734 indicates the location oflocalization element 660 of percutaneous needle 650. In yet otherembodiments, for example, indicia 734 indicates the location of tip 657of percutaneous needle 650. An actual trajectory 736 of percutaneousneedle 650 is also displayed on display 80 by navigation system 70 asshown in panels 724, 726. As described more fully elsewhere herein,suggested trajectory 730 may avoid anatomical structures 740 such as,for example, bone, the heart, the liver, other organs, fissures,diseased tissue, such as chronic obstructive pulmonary disease (COPD)lung tissue, and blood vessels. Furthermore, as shown in panel 724,navigation system 70 may be able to display a distance from tip 657 ofpercutaneous needle 650 to the confirmed location of the target tissue.

Referring again to FIG. 20C, at step 1022, the physician or otherhealthcare professional inserts percutaneous needle 650 into the patientand navigates tip 657 proximate to the confirmed location of the targettissue. Then at step 1024, the target tissue at the confirmed locationis intercepted. In certain embodiments, for example, intercepting thetarget tissue at the confirmed location includes inserting a biopsydevice into working channel 658 of percutaneous needle 650 and extendingthe biopsy device beyond tip 657 to sample the target tissue. In otherembodiments, for example, intercepting the target tissue at theconfirmed location includes inserting a therapy device into workingchannel 658 of percutaneous needle 650 and delivering therapy to thetarget tissue. In various embodiments, therapy device may be an ablationprobe and navigation system 70 may be able to display on display 80ablation models at the confirmed location of the target tissue. Theablation models may assist the physician or other healthcareprofessional in delivering the appropriate amount of treatment to thetarget tissue. The method may optionally continue according to stepsillustrated in FIG. 20D as described more fully elsewhere herein.

In various embodiments, the method as described in FIGS. 20A-20C mayfurther include the step of taking a population of images of at least aportion of percutaneous needle 650 at the confirmed location of thetarget tissue using imaging device 633 disposed in the airway of thepatient. For example, as described above, imaging device 633 may be EBUSdevice 634 extended out tip 607 of steerable catheter 600. The imagesmay be used to confirm that tip 657 of percutaneous needle 650 wasactually navigated to proximate the confirmed location of the targettissue. The image(s) of percutaneous needle 650 at the confirmedlocation of the target tissue may be recorded into a patient file asproof that the confirmed location of the target was reached.Additionally, imaging device 633 may be used to generate a population ofimages of the biopsy device sampling the target tissue and/or apopulation of images of the therapy device delivering therapy to thetarget tissue. The image(s) of biopsy device and therapy device samplingor delivering therapy to the target tissue may be recorded into apatient file as proof that the target tissue was sampled and/or treated.

Additional to or alternative to using imaging device 633 to evaluatewhether percutaneous needle 650 has been navigated to proximate theconfirmed location of the target tissue, a sensing device may be used tosense the presence of at least a portion of percutaneous needle 650 atthe confirmed location of the target tissue. For example, the sensingdevice may include, but is not limited to, a heat sensor, magneticsensor, electrical sensor, that may be extended out tip 607 of steerablecatheter 600. In certain embodiments, the sensing device may also beable to sense the presence of the biopsy device sampling the targettissue and/or the therapy device delivering therapy to the targettissue. For example, a heat sensor extended out tip 607 of steerablecatheter 600 may be used to determine when the target tissue has beensufficiently treated. Additionally, navigating steerable catheter 600down multiple airways adjacent to a target tissue and extending a heatsensor out tip 607 of steerable catheter 600 in each of the adjacentairways may be used to determine when a target tissue that is locatedbetween the adjacent airways has been treated. In certain embodiments,heat sensors may be placed in multiple airways adjacent to a targettissue using steerable catheter 600 and the multiple heat sensors may beused to determine when a target tissue that is located between theadjacent airways has been treated.

In various embodiments, the method as described in FIGS. 20A-20C mayfurther include the steps outlined in FIG. 20D. At step 1026, usingimaging device 633 disposed in the airway of patient 10, a population ofimages are generated of one or more anatomical structures proximate theconfirmed location of the target tissue. Anatomical structures mayinclude, but are not limited to, bone, the heart, the liver, otherorgans, fissures, diseased tissue, such as, for example chronicobstructive pulmonary disease (COPD) lung tissue, and blood vessels.Accordingly, the anatomical structures may be any structure within thebody of patient 10 that should be avoided, if possible, by percutaneousneedle 650. The imaging device, for example, may be EBUS device 634extended out tip 607 of steerable catheter 600. At step 1028, aconfirmed location of the anatomical structure(s) is determined inrelation to the location of PTD 20 using the population of images andthe tracked location of localization element 610 of steerable catheter600. For example, navigation system 70 tracks the extension (x), if any,of EBUS device 634 in relation to localization element 610. By trackingthe extension (x) in relation to localization element 610, navigationsystem 70 knows the coordinates at which the population of images of theanatomical structure(s) are generated and may thus determine the actuallocation and size of the anatomical structure(s) within patient 10 withrespect to PTD 20.

At step 1030, after the location of the anatomical structure(s) isdetermined, the confirmed location of the anatomical structure(s) isrecorded. In one embodiment, for example, recording the confirmedlocation of the anatomical structure(s) comprises recording athree-dimensional (3D) location of the confirmed anatomical structure(s)in relation to PTD 20. In another embodiment, for example, recording theconfirmed location of the anatomical structure(s) comprises recording athree-dimensional (3D) location of the confirmed anatomical structure(s)in relation to electromagnetic (EM) field generator 82 of navigationsystem 70. In one embodiment, for example, recording the confirmedlocation of the anatomical structure(s) comprises recordingfour-dimensional data (4D) comprising a three-dimensional (3D) locationof the confirmed anatomical structure(s) in relation to PTD 20 and therespiratory state of patient 10 at the time the location of theanatomical structure(s) was confirmed. In another embodiment, forexample, recording the confirmed location of the anatomical structure(s)comprises recording four-dimensional data (4D) comprising athree-dimensional (3D) location of the confirmed anatomical structure(s)in relation to electromagnetic (EM) field generator 82 of navigationsystem 70 and the respiratory state of patient 10 at the time thelocation of the anatomical structure(s) was confirmed. In yet anotherembodiment, for example, recording the confirmed location of theanatomical structure(s) comprises recording four-dimensional (4D) datacomprising a three-dimensional location (3D) of the confirmed anatomicalstructure(s) in relation to PTD 20 and a cardiac state of the patient atthe time the location of the anatomical structure(s) was confirmed. Inyet another embodiment, for example, recording the confirmed location ofthe anatomical structure(s) comprises recording four-dimensional (4D)data comprising a three-dimensional location (3D) of the confirmedanatomical structure(s) in relation to electromagnetic (EM) fieldgenerator 82 and a cardiac state of the patient at the time the locationof the anatomical structure(s) was confirmed. In various embodiments,this confirmed location of the anatomical structure(s) may then beapplied to one or more images from image dataset 400 depicting theairway at the respiratory state of patient 10 at the time the locationof the anatomical structure(s) was confirmed. This information isrecorded in memory component 74 of navigation system 70.

Optionally, at step 1032, navigation system 70 calculates and displays atrajectory of a percutaneous device (e.g., percutaneous needle 650) fromthe confirmed location of the target tissue to a corresponding entrypoint on the body of patient 10. This trajectory may avoid some or allof the anatomical structures. Accordingly, if a physician or otherhealthcare professional inserts percutaneous device, such aspercutaneous needle 650, following this trajectory, the percutaneousdevice may avoid some or all of the anatomical structures therebypreventing damage to the anatomical structure(s).

In various embodiments, in addition to calculating and/or displaying anyof the trajectories described herein, navigation system 70 displays anextended trajectory of a medical device that may be inserted intoworking channel 658 of percutaneous needle 650 and extended past tip657. In certain embodiments, for example, the medical device mayinclude, but is not limited to, an aspiration needle, a forceps device,a brush, or any type of biopsy device. In other embodiments, forexample, the medical device may include, but is not limited to, anablation probe, a radioactive seed placement device, a fiducialplacement device, and/or any type of therapy device. The extendedtrajectory displays the potential extension of the medical device sothat it may be confirmed that potential extension of the medical devicewill sample and/or treat the target tissue and will not hit one or moreanatomical structures. The displayed extended trajectory may also aid inensuring that a sufficient sample is taken and/or that the treatment maybe properly placed in the target tissue.

In various embodiments, the method as described in FIGS. 20A-20B mayfurther include the steps outlined in FIG. 20E. In addition to oralternative to generating images of anatomical structures of patient 10using imaging device 633 inserted into the airway of patient 10, one ormore atlas models are employed to assist the procedure during the secondtime interval. The atlas model(s) are three-dimensional models of humananatomy and therefore include a variety of anatomical structures. Theanatomical structures may include, but are not limited to, bone, theheart, the liver, other organs, fissures, diseased tissue, such as, forexample, chronic obstructive pulmonary disease (COPD) lung tissue, andblood vessels. Accordingly, the anatomical structures may be anystructure within the body of patient 10 that should be avoided, ifpossible, by percutaneous device (e.g., percutaneous needle 650).Additionally, the atlas model(s) may include weighted informationrelated to the acceptability of a planned trajectory or planned ablationprocedure to determine the optimal plan. This weighted information mayinclude, but is not limited to, information regarding which anatomicalstructure(s) cannot be crossed by a medical device, informationregarding avoid anatomical structure(s) by at least a given distance,and information regarding the heat sink effect of anatomicalstructure(s) so that ablation location and amount may be adjusted.

Thus as shown in FIG. 20E at step 1034, one or more atlas models ismatched to image dataset 400 of patient 10 wherein the matching maycomprise deforming the atlas model(s) to the image dataset 400 and/orregistering the atlas model(s) to patient 10. At step 1036, navigationsystem 70 identifies anatomical structure(s) to be avoided by thetrajectory of the percutaneous device. At step 1038, navigation system70 may calculate and display a trajectory of the percutaneous devicefrom the confirmed location of the target tissue to a correspondingentry point on the body of patient 10. This trajectory may avoid some orall of the anatomical structures. Accordingly, if a physician or otherhealthcare professional inserts percutaneous device, such aspercutaneous needle 650, following this trajectory, percutaneous devicemay avoid some or all of the anatomical structures thereby preventingdamage to the anatomical structure(s). Following the steps outlined inFIG. 20E, the method may optionally further include the stepsillustrated in FIG. 20C.

In any of the embodiments of the methods described herein, a dye may beinjected into the target tissue at the confirmed location using a needleinserted into working channel 608 of steerable catheter 600 or using aneedle inserted into working channel 658 of percutaneous needle 650.Thus, when sampling the target tissue using a medical device insertedinto working channel 658 of percutaneous needle 650, the presence of dyein the sample provides another indication that the correct target tissuewas sampled. These additional steps may be helpful, for example, in lungresections where there is significant movement of the lungs of patient10. For example, during lung resections there may be a gap between thechest wall and the lung and the physician or other healthcare professionmay use a rigid scope to enter into patient 10. Because the confirmedtarget tissue was previously dyed using a needle inserted into workingchannel 608 of steerable catheter 600 or using a needle inserted intoworking channel 658 of percutaneous needle 650, the physician or otherhealthcare professional may be able to visually see the dye. This mayassist the physician or healthcare professional in sampling and/ortreating the correct target tissue.

Additionally, in various embodiments of the methods described herein,after tip 607 of steerable catheter 600 has been navigated proximateconfirmed location of target tissue a sample of air proximate theconfirmed location of the target tissue may be taken. Then cells, scentsor other potential indicators of cancer within the air sample may thenbe analyzed to determine if the target tissue is cancerous. In certainembodiments, a breath analysis device may be inserted into workingchannel 608 of steerable catheter 600 and this breath analysis devicemay sample the air in situ. In other embodiments, a vacuum of air may bedrawn on working channel 608 from port 616 to sample the air proximatethe confirmed location of the target tissue may be taken. The vacuum maybe created by a syringe inserted into port 616 or by some other suctiondevice known in the art. In yet other embodiments, a sample of airproximate an airway segment near the confirmed location of the targettissue may be taken instead of, or in addition to, the sample takenproximate the confirmed location of the target.

Furthermore, in any of the embodiments of the methods described herein,navigation system 70 may be able to control a robotic medical devicehaving a percutaneous needle. Navigation system 70 may be able to causerobotic medical device to navigate a percutaneous needle to thecalculated entry point on the surface of patient 10. The percutaneousneedle may then be inserted into patient 10 at the calculated entrypoint on the surface of patient 10 and the percutaneous needle may beextended to the confirmed location along the calculated trajectory.Thus, a robotic medical device may use information from navigationsystem 70 to perform any of the methods described herein.

The accompanying Figures and this description depict and describecertain embodiments of a navigation system (and related methods anddevices) in accordance with the present invention, and features andcomponents thereof. It should also be noted that any references hereinto front and back, right and left, top and bottom and upper and lowerare intended for convenience of description, not to limit the presentinvention or its components to any one positional or spatialorientation.

It is noted that the terms “comprise” (and any form of comprise, such as“comprises” and “comprising”), “have” (and any form of have, such as“has” and “having”), “contain” (and any form of contain, such as“contains” and “containing”), and “include” (and any form of include,such as “includes” and “including”) are open-ended linking verbs. Thus,a method, an apparatus, or a system that “comprises,” “has,” “contains,”or “includes” one or more items possesses at least those one or moreitems, but is not limited to possessing only those one or more items.Individual elements or steps of the present methods, apparatuses, andsystems are to be treated in the same manner.

The terms “a” and “an” are defined as one or more than one. The term“another” is defined as at least a second or more. The term “coupled”encompasses both direct and indirect connections, and is not limited tomechanical connections.

Those of skill in the art will appreciate that in the detaileddescription above, certain well known components and assembly techniqueshave been omitted so that the present methods, apparatuses, and systemsare not obscured in unnecessary detail.

While various embodiments of the invention have been described above, itshould be understood that they have been presented by way of exampleonly, and not limitation. Thus, the breadth and scope of the inventionshould not be limited by any of the above-described embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

The previous description of the embodiments is provided to enable anyperson skilled in the art to make or use the invention. While theinvention has been particularly shown and described with reference toembodiments thereof, it will be understood by those skilled in art thatvarious changes in form and details may be made therein withoutdeparting from the spirit and scope of the invention. For example, thepatient tracking device, steerable catheter, percutaneous needle, andlocalization elements may be constructed from any suitable material, andmay be a variety of different shapes and sizes, not necessarilyspecifically illustrated, while still remaining within the scope of theinvention.

The invention claimed is:
 1. A method of orienting an image feed on adisplay comprising: a) inserting an imaging device that provides theimage feed into a channel of a steerable catheter, wherein the steerablecatheter has a distal end portion and a localization element proximatethe distal end portion; b) using the imaging device to generate areference image feed of a reference of a known orientation; c) using aprocessor to determine a registered rotation angle θ by which thereference image feed must be rotated to obtain a registered orientationsuch that the reference in the reference image feed is matched to itsknown orientation when the reference image feed is rotated to theregistered orientation; d) determining a localization elementorientation of the localization element of the steerable catheter; e)determining a localization element angle β between the registeredorientation and the localization element orientation; and f) using theprocessor and the determined localization element angle β to ensure thata real-time image feed from the imaging device is shown in an “up”orientation on the display as the steerable catheter is manipulated. 2.The method of claim 1, wherein the processor and the display form partof a navigation system, and further wherein the steerable catheter isnavigated into an airway of a patient using the navigation.
 3. Themethod of claim 1, wherein the step of using the processor to determinethe registered rotation angle θ further comprises ensuring that nosteering input is applied to the steerable catheter after generating thereference image feed of the reference while using the processor todetermine the registered rotation angle θ.
 4. The method of claim 2,wherein the reference comprises an anatomical feature of the airway ofthe patient.
 5. The method of claim 2, further comprising overlaying anavigational aid onto the real-time image feed, wherein the navigationalaid is registered to the real-time image feed.
 6. The method of claim 2,further comprising displaying a virtual volumetric scene of the airwayon the display and registering the virtual volumetric scene to thereal-time image feed.
 7. The method of claim 2, wherein the localizationelement orientation is determined by using the navigation system todetermine the position and orientation of the localization element. 8.The method of claim 5, wherein the navigational aid is selected from aset of aids comprising a navigation pathway and a directional cue.